Question:
Calculate Time Difference Between Two Pandas Columns in Hours and Minutes I have two columns, fromdate and todate, in a dataframe.
```py
import pandas as pd

data = {'todate': [pd.Timestamp('2014-01-24 13:03:12.050000'), pd.Timestamp('2014-01-27 11:57:18.240000'), pd.Timestamp('2014-01-23 10:07:47.660000')],
        'fromdate': [pd.Timestamp('2014-01-26 23:41:21.870000'), pd.Timestamp('2014-01-27 15:38:22.540000'), pd.Timestamp('2014-01-23 18:50:41.420000')]}

df = pd.DataFrame(data)
```
I add a new column, diff, to find the difference between the two dates using
```py
df['diff'] = df['fromdate'] - df['todate']
```
I get the diff column, but it contains days, when there's more than 24 hours.
```none
todate                 fromdate                    diff
0 2014-01-24 13:03:12.050  2014-01-26 23:41:21.870  2 days 10:38:09.820000
1 2014-01-27 11:57:18.240  2014-01-27 15:38:22.540  0 days 03:41:04.300000
2 2014-01-23 10:07:47.660  2014-01-23 18:50:41.420  0 days 08:42:53.760000
```
How do I convert my results to only hours and minutes (i.e. days are converted to hours)?

Response:
Pandas timestamp differences returns a datetime.timedelta object. This can easily be converted into hours by using the *as_type* method, like so
```
import pandas
df = pandas.DataFrame(columns=['to','fr','ans'])
df.to = [pandas.Timestamp('2014-01-24 13:03:12.050000'), pandas.Timestamp('2014-01-27 11:57:18.240000'), pandas.Timestamp('2014-01-23 10:07:47.660000')]
df.fr = [pandas.Timestamp('2014-01-26 23:41:21.870000'), pandas.Timestamp('2014-01-27 15:38:22.540000'), pandas.Timestamp('2014-01-23 18:50:41.420000')]
(df.fr-df.to).astype('timedelta64[h]')
```
to yield,
```
0    58
1     3
2     8
dtype: float64
```


Question:
Convert categorical data in pandas dataframe I have a dataframe with this type of data (too many columns):
```
col1        int64
col2        int64
col3        category
col4        category
col5        category
```
Columns look like this:
```
Name: col3, dtype: category
Categories (8, object): [B, C, E, G, H, N, S, W]
```
I want to convert all the values in each column to integer like this:
```
[1, 2, 3, 4, 5, 6, 7, 8]
```
I solved this for one column by this:
```
dataframe['c'] = pandas.Categorical.from_array(dataframe.col3).codes
```
Now I have two columns in my dataframe - old col3 and new c and need to drop old columns. That's bad practice. It works but in my dataframe there are too many columns and I don't want do it manually. How can I do this more cleverly?

Response:
First, to convert a Categorical column to its numerical codes, you can do this easier with: dataframe['c'].cat.codes. Further, it is possible to select automatically all columns with a certain dtype in a dataframe using select_dtypes. This way, you can apply above operation on multiple and automatically selected columns. First making an example dataframe:
```

```python
df = pd.DataFrame({'col1':[1,2,3,4,5], 'col2':list('abcab'),  'col3':list('ababb')})
```

```python
df['col2'] = df['col2'].astype('category')
```

```python
df['col3'] = df['col3'].astype('category')
```

```python
df.dtypes
#Output
#col1       int64
#col2    category
#col3    category
#dtype: object
#```
#Then by using select_dtypes to select the columns, and then applying .cat.codes on each of these columns, you can get the following result:
#```
```

```python
cat_columns = df.select_dtypes(['category']).columns
```

```python
cat_columns
#Output
#Index([u'col2', u'col3'], dtype='object')
```

```python
df[cat_columns] = df[cat_columns].apply(lambda x: x.cat.codes)
```

```python
df
#Output
#col1  col2  col3
#0     1     0     0
#1     2     1     1
#2     3     2     0
#3     4     0     1
#4     5     1     1
#```
#Note: NaN becomes -1 This method is fast because the relationship between code and category is readily available and do not need to be computed.
```


Question:
Creating a zero-filled pandas data frame What is the best way to create a zero-filled pandas data frame of a given size? I have used:
```
zero_data = np.zeros(shape=(len(data),len(feature_list)))
d = pd.DataFrame(zero_data, columns=feature_list)
```
Is there a better way to do it?

Response:
Create and fill a pandas dataframe with zeros
```
feature_list = ["foo", "bar", 37]
df = pd.DataFrame(0, index=np.arange(7), columns=feature_list)
print(df)
```
which prints:
```
foo  bar  37
0    0    0   0
1    0    0   0
2    0    0   0
3    0    0   0
4    0    0   0
5    0    0   0
6    0    0   0
```


Question:
how to test if a variable is pd.NaT? I'm trying to test if one of my variables is pd.NaT. I know it is NaT, and still it won't pass the test. As an example, the following code prints nothing :
```
a=pd.NaT

if a == pd.NaT:
    print("a not NaT")
```
Does anyone have a clue ? Is there a way to effectively test if a is NaT?

Response:
Pandas NaT behaves like a floating-point NaN, in that it's not equal to itself. Instead, you can use pandas.isnull:
```

```python
pandas.isnull(pandas.NaT)
#Output
#True
#```
#This also returns True for None and NaN. Technically, you could also check for Pandas NaT with x != x, following a common pattern used for floating-point NaN. However, this is likely to cause issues with NumPy NaTs, which look very similar and represent the same concept, but are actually a different type with different behavior:
#```
```

```python
x = pandas.NaT
```

```python
y = numpy.datetime64('NaT')
```

```python
x != x
#Output
#True
```

```python
y != y
/home/i850228/.local/lib/python3.6/site-packages/IPython/__main__.py:1: FutureWarning: In the future, NAT != NAT will be True rather than False.
  # encoding: utf-8
#Output
#False
#```
#numpy.isnat, the function to check for NumPy NaT, also fails with a Pandas NaT:
#```
```

```python
numpy.isnat(pandas.NaT)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-33-39a66bbf6513> in <module>()
----> 1 numpy.isnat(pandas.NaT)

TypeError: ufunc 'isnat' is only defined for datetime and timedelta.
```
pandas.isnull works for both Pandas and NumPy NaTs, so it's probably the way to go:
```
```

```python
pandas.isnull(pandas.NaT)
#Output
#True
```

```python
pandas.isnull(numpy.datetime64('NaT'))
#Output
#True
#```
```


Question: How to convert single-row pandas data frame to series? I'm somewhat new to pandas. I have a pandas data frame that is 1 row by 23 columns. I want to convert this into a series. I'm wondering what the most pythonic way to do this is? I've tried pd.Series(myResults) but it complains ValueError: cannot copy sequence with size 23 to array axis with dimension 1. It's not smart enough to realize it's still a "vector" in math terms.

Response:
You can transpose the single-row dataframe (which still results in a dataframe) and then squeeze the results into a series (the inverse of to_frame).
```
df = pd.DataFrame([list(range(5))], columns=["a{}".format(i) for i in range(5)])

```python
df.squeeze(axis=0)
#Output
#a0    0
#a1    1
#a2    2
#a3    3
#a4    4
#Name: 0, dtype: int64
#```
#Note: To accommodate the point raised by @IanS (even though it is not in the OP's question), test for the dataframe's size. I am assuming that df is a dataframe, but the edge cases are an empty dataframe, a dataframe of shape (1, 1), and a dataframe with more than one row in which case the use should implement their desired functionality.
#```
#if df.empty:
#    # Empty dataframe, so convert to empty Series.
#    result = pd.Series()
#elif df.shape == (1, 1)
#    # DataFrame with one value, so convert to series with appropriate index.
#    result = pd.Series(df.iat[0, 0], index=df.columns)
#elif len(df) == 1:
#    # Convert to series per OP's question.
#    result = df.T.squeeze()
#else:
#    # Dataframe with multiple rows.  Implement desired behavior.
#    pass
#```
#This can also be simplified along the lines of the answer provided by @themachinist.
#```
#if len(df) > 1:
#    # Dataframe with multiple rows.  Implement desired behavior.
#    pass
#else:
#    result = pd.Series() if df.empty else df.iloc[0, :]
#```
```


Question:
pandas - add new column to dataframe from dictionary [duplicate] This question already has answers here: Remap values in pandas column with a dict, preserve NaNs (12 answers) Closed 6 years ago. I would like to add a column 'D' to a dataframe like this:
```
U,L
111,en
112,en
112,es
113,es
113,ja
113,zh
114,es
```
based on the following Dictionary:
```
d = {112: 'en', 113: 'es', 114: 'es', 111: 'en'}
```
so that the resulting dataframe appears as:
```
U,L,D
111,en,en
112,en,en
112,es,en
113,es,es
113,ja,es
113,zh,es
114,es,es
```
So far I tried the pd.join() method but I can't figured out how it works with Dictionaries.

Response:
Call map and pass the dict, this will perform a lookup and return the associated value for that key:
```

```python
d = {112: 'en', 113: 'es', 114: 'es', 111: 'en'}
df['D'] = df['U'].map(d)
df
#Output
#U   L   D
#0  111  en  en
#1  112  en  en
#2  112  es  en
#3  113  es  es
#4  113  ja  es
#5  113  zh  es
#6  114  es  es
#```
```


Question:
Append column to pandas dataframe This is probably easy, but I have the following data: In data frame 1:
```
index dat1
0     9
1     5
```
In data frame 2:
```
index dat2
0     7
1     6
```
I want a data frame with the following form:
```
index dat1  dat2
0     9     7
1     5     6
```
I've tried using the append method, but I get a cross join (i.e. cartesian product). What's the right way to do this?

Response:
It seems in general you're just looking for a join:
```
> dat1 = pd.DataFrame({'dat1': [9,5]})
> dat2 = pd.DataFrame({'dat2': [7,6]})
> dat1.join(dat2)
   dat1  dat2
0     9     7
1     5     6
```


Question:
Joining pandas DataFrames by Column names I have two DataFrames with the following column names:
```none
frame_1:
event_id, date, time, county_ID

frame_2:
countyid, state
```
I would like to get a DataFrame with the following columns by left-joining on county_ID = countyid:
```none
joined_dataframe:
event_id, date, time, county, state
```
I cannot figure out how to do it if the columns on which I want to join are not the index.

Response:
You can use the left_on and right_on options of pd.merge as follows:
```
pd.merge(frame_1, frame_2, left_on='county_ID', right_on='countyid')
```
Or equivalently with DataFrame.merge:
```
frame_1.merge(frame_2, left_on='county_ID', right_on='countyid')
```
I was not sure from the question if you only wanted to merge if the key was in the left hand DataFrame. If that is the case then the following will do that (the above will in effect do a many to many merge)
```
pd.merge(frame_1, frame_2, how='left', left_on='county_ID', right_on='countyid')
```
Or
```
frame_1.merge(frame_2, how='left', left_on='county_ID', right_on='countyid')
```


Question: Adding meta-information/metadata to pandas DataFrame Is it possible to add some meta-information/metadata to a pandas DataFrame? For example, the instrument's name used to measure the data, the instrument responsible, etc. One workaround would be to create a column with that information, but it seems wasteful to store a single piece of information in every row!

Response:
Sure, like most Python objects, you can attach new attributes to a pandas.DataFrame:
```
import pandas as pd
df = pd.DataFrame([])
df.instrument_name = 'Binky'
```
Note, however, that while you can attach attributes to a DataFrame, operations performed on the DataFrame (such as groupby, pivot, join, assign or loc to name just a few) may return a new DataFrame without the metadata attached. Pandas does not yet have a robust method of propagating metadata attached to DataFrames. Preserving the metadata in a file is possible. You can find an example of how to store metadata in an HDF5 file here.


Question:
Apply function to each cell in DataFrame I have a dataframe that may look like this:
```none
A        B        C
foo      bar      foo bar
bar foo  foo      bar
```
I want to look through every element of each row (or every element of each column) and apply the following function to get the subsequent dataframe:
```py
def foo_bar(x):
    return x.replace('foo', 'wow')
```
After applying the function, my dataframe will look like this:
```none
A        B        C
wow      bar      wow bar
bar wow  wow      bar
```
Is there a simple one-liner that can apply a function to each cell? This is a simplistic example so there may be an easier way to execute this specific example other than applying a function, but what I am really asking about is how to apply a function in every cell within a dataframe.

Response:
You can use map() which is concise for your case.
```
df.map(foo_bar)

#     A       B       C
#0  wow     bar wow bar
#1  bar wow wow     bar
```
Another option is to vectorize your function and then use apply method:
```
import numpy as np
df.apply(np.vectorize(foo_bar))
#     A       B       C
#0  wow     bar wow bar
#1  bar wow wow     bar
```


Question:
Pandas "Can only compare identically-labeled DataFrame objects" error I'm using Pandas to compare the outputs of two files loaded into two data frames (uat, prod): ...
```
uat = uat[['Customer Number','Product']]
prod = prod[['Customer Number','Product']]
print uat['Customer Number'] == prod['Customer Number']
print uat['Product'] == prod['Product']
print uat == prod

The first two match exactly:
74357    True
74356    True
Name: Customer Number, dtype: bool
74357    True
74356    True
Name: Product, dtype: bool
```
For the third print, I get an error: Can only compare identically-labeled DataFrame objects. If the first two compared fine, what's wrong with the 3rd? Thanks

Response:
Here's a small example to demonstrate this (which only applied to DataFrames, not Series, until Pandas 0.19 where it applies to both):
```

```python
df1 = pd.DataFrame([[1, 2], [3, 4]])
```

```python
df2 = pd.DataFrame([[3, 4], [1, 2]], index=[1, 0])
```

```python
df1 == df2
Exception: Can only compare identically-labeled DataFrame objects
```
One solution is to sort the index first (Note: some functions require sorted indexes):
```
```

```python
df2.sort_index(inplace=True)
```

```python
df1 == df2
#Output
#0     1
#0  True  True
#1  True  True
#```
#Note: == is also sensitive to the order of columns, so you may have to use sort_index(axis=1):
#```
```

```python
df1.sort_index().sort_index(axis=1) == df2.sort_index().sort_index(axis=1)
#Output
#0     1
#0  True  True
#1  True  True
#```
#Note: This can still raise (if the index/columns aren't identically labelled after sorting).
```


Question:
Remove index name in pandas I have a dataframe like this one:
```

```python
df
#Output
#Column 1
#foo
#Apples          1
#Oranges         2
#Puppies         3
#Ducks           4
#```
#How to remove index name foo from that dataframe? The desired output is like this:
#```
```

```python
df
#Output
#Column 1
#Apples          1
#Oranges         2
#Puppies         3
#Ducks           4
#```
```

Response:
Took me way too long to find an answer that actually worked for me. See below.
```
df = df.rename_axis(None, axis=1)
```
I'm sure some of these other answers are working for other people, but they definitely didn't work for me :(


Question:
Extract first and last row of a dataframe in pandas How can I extract the first and last rows of a given dataframe as a new dataframe in pandas? I've tried to use iloc to select the desired rows and then concat as in:
```
df=pd.DataFrame({'a':range(1,5), 'b':['a','b','c','d']})
pd.concat([df.iloc[0,:], df.iloc[-1,:]])
```
but this does not produce a pandas dataframe:
```
a    1
b    a
a    4
b    d
dtype: object
```

Response:
I think the most simple way is .iloc[[0, -1]].
```
df = pd.DataFrame({'a':range(1,5), 'b':['a','b','c','d']})
df2 = df.iloc[[0, -1]]

print(df2)

   a  b
0  1  a
3  4  d
```


Question:
Does pandas iterrows have performance issues? I have noticed very poor performance when using iterrows from pandas. Is it specific to iterrows and should this function be avoided for data of a certain size (I'm working with 2-3 million rows)? This discussion on GitHub led me to believe it is caused when mixing dtypes in the dataframe, however the simple example below shows it is there even when using one dtype (float64). This takes 36 seconds on my machine:
```
import pandas as pd
import numpy as np
import time

s1 = np.random.randn(2000000)
s2 = np.random.randn(2000000)
dfa = pd.DataFrame({'s1': s1, 's2': s2})

start = time.time()
i=0
for rowindex, row in dfa.iterrows():
    i+=1
end = time.time()
print end - start
```
Why are vectorized operations like apply so much quicker? I imagine there must be some row by row iteration going on there too. I cannot figure out how to not use iterrows in my case (this I'll save for a future question). Therefore I would appreciate hearing if you have consistently been able to avoid this iteration. I'm making calculations based on data in separate dataframes. A simplified version of what I want to run:
```
import pandas as pd
import numpy as np

#%% Create the original tables
t1 = {'letter':['a','b'],
      'number1':[50,-10]}

t2 = {'letter':['a','a','b','b'],
      'number2':[0.2,0.5,0.1,0.4]}

table1 = pd.DataFrame(t1)
table2 = pd.DataFrame(t2)

#%% Create the body of the new table
table3 = pd.DataFrame(np.nan, columns=['letter','number2'], index=[0])

#%% Iterate through filtering relevant data, optimizing, returning info
for row_index, row in table1.iterrows():
    t2info = table2[table2.letter == row['letter']].reset_index()
    table3.ix[row_index,] = optimize(t2info,row['number1'])

#%% Define optimization
def optimize(t2info, t1info):
    calculation = []
    for index, r in t2info.iterrows():
        calculation.append(r['number2']*t1info)
    maxrow = calculation.index(max(calculation))
    return t2info.ix[maxrow]
```

Response: Generally, iterrows should only be used in very, very specific cases. This is the general order of precedence for performance of various operations: vectorization using a custom Cython routine apply reductions that can be performed in Cython iteration in Python space itertuples iterrows updating an empty frame (e.g., using loc one-row-at-a-time) Using a custom Cython routine is usually too complicated, so let's skip that for now. Vectorization is always, always the first and best choice. However, there is a small set of cases (usually involving a recurrence) which cannot be vectorized in obvious ways. Furthermore, on a smallish DataFrame, it may be faster to use other methods. apply usually can be handled by an iterator in Cython space. This is handled internally by pandas, though it depends on what is going on inside the apply expression. For example, df.apply(lambda x: np.sum(x)) will be executed pretty swiftly, though of course, df.sum(1) is even better. However something like df.apply(lambda x: x['b'] + 1) will be executed in Python space, and consequently is much slower. itertuples does not box the data into a Series. It just returns the data in the form of tuples. iterrows does box the data into a Series. Unless you really need this, use another method. Updating an empty frame a-single-row-at-a-time. I have seen this method used WAY too much. It is by far the slowest. It is probably common place (and reasonably fast for some Python structures), but a DataFrame does a fair number of checks on indexing, so this will always be very slow to update a row at a time. Much better to create new structures and concat.


Question:
Selecting pandas column by location I'm simply trying to access named pandas columns by an integer. You can select a row by location using df.ix[3]. But how to select a column by integer? My dataframe:
```
df=pandas.DataFrame({'a':np.random.rand(5), 'b':np.random.rand(5)})
```

Response:
Two approaches that come to mind:
```

```python
df
#Output
#          A         B         C         D
#0  0.424634  1.716633  0.282734  2.086944
#1 -1.325816  2.056277  2.583704 -0.776403
#2  1.457809 -0.407279 -1.560583 -1.316246
#3 -0.757134 -1.321025  1.325853 -2.513373
#4  1.366180 -1.265185 -2.184617  0.881514
```

```python
df.iloc[:, 2]
#Output
#0    0.282734
#1    2.583704
#2   -1.560583
#3    1.325853
#4   -2.184617
#Name: C
```

```python
df[df.columns[2]]
#Output
#0    0.282734
#1    2.583704
#2   -1.560583
#3    1.325853
#4   -2.184617
#Name: C
#```
#Edit: The original answer suggested the use of df.ix[:,2] but this function is now deprecated. Users should switch to df.iloc[:,2].
```


Question:
What is the difference between NaN and None? I am reading two columns of a csv file using pandas readcsv() and then assigning the values to a dictionary. The columns contain strings of numbers and letters. Occasionally there are cases where a cell is empty. In my opinion, the value read to that dictionary entry should be None but instead nan is assigned. Surely None is more descriptive of an empty cell as it has a null value, whereas nan just says that the value read is not a number. Is my understanding correct, what IS the difference between None and nan? Why is nan assigned instead of None? Also, my dictionary check for any empty cells has been using numpy.isnan():
```
for k, v in my_dict.iteritems():
    if np.isnan(v):
```
But this gives me an error saying that I cannot use this check for v. I guess it is because an integer or float variable, not a string is meant to be used. If this is true, how can I check v for an "empty cell"/nan case?

Response:
NaN is used as a placeholder for missing data consistently in pandas, consistency is good. I usually read/translate NaN as "missing". Also see the 'working with missing data' section in the docs. Wes writes in the docs 'choice of NA-representation': After years of production use [NaN] has proven, at least in my opinion, to be the best decision given the state of affairs in NumPy and Python in general. The special value NaN (Not-A-Number) is used everywhere as the NA value, and there are API functions isna and notna which can be used across the dtypes to detect NA values. ... Thus, I have chosen the Pythonic “practicality beats purity” approach and traded integer NA capability for a much simpler approach of using a special value in float and object arrays to denote NA, and promoting integer arrays to floating when NAs must be introduced. Note: the "gotcha" that integer Series containing missing data are upcast to floats. In my opinion the main reason to use NaN (over None) is that it can be stored with numpy's float64 dtype, rather than the less efficient object dtype, see NA type promotions.
```
#  without forcing dtype it changes None to NaN!
s_bad = pd.Series([1, None], dtype=object)
s_good = pd.Series([1, np.nan])

```python
s_bad.dtype
#Output
#dtype('O')
```

```python
s_good.dtype
#Output
#dtype('float64')
#```
#Jeff comments (below) on this: np.nan allows for vectorized operations; its a float value, while None, by definition, forces object type, which basically disables all efficiency in numpy. So repeat 3 times fast: object==bad, float==good Saying that, many operations may still work just as well with None vs NaN (but perhaps are not supported i.e. they may sometimes give surprising results):
#```
```

```python
s_bad.sum()
#Output
#1
```

```python
s_good.sum()
#Output
#1.0
#```
#To answer the second question: You should be using isna and notna to test for missing data (NaN).
```


Question:
Convert number strings with commas in pandas DataFrame to float I have a DataFrame that contains numbers as strings with commas for the thousands marker. I need to convert them to floats.
```
a = [['1,200', '4,200'], ['7,000', '-0.03'], [ '5', '0']]
df=pandas.DataFrame(a)
```
I am guessing I need to use locale.atof. Indeed
```
df[0].apply(locale.atof)
```
works as expected. I get a Series of floats. But when I apply it to the DataFrame, I get an error.
```
df.apply(locale.atof)
```
TypeError: ("cannot convert the series to ", u'occurred at index 0') and
```
df[0:1].apply(locale.atof)
```
gives another error: ValueError: ('invalid literal for float(): 1,200', u'occurred at index 0') So, how do I convert this DataFrame of strings to a DataFrame of floats?

Response:
If you're reading in from csv then you can use the thousands arg:
```
df.read_csv('foo.tsv', sep='\t', thousands=',')
```
This method is likely to be more efficient than performing the operation as a separate step. You need to set the locale first:
```
In [ 9]: import locale

```python
from locale import atof
```

```python
locale.setlocale(locale.LC_NUMERIC, '')
#Output
#'en_GB.UTF-8'
```

```python
df.applymap(atof)
#Output
#0        1
#0  1200  4200.00
#1  7000    -0.03
#2     5     0.00
#```
```


Question:
Reading tab-delimited file with Pandas - works on Windows, but not on Mac I've been reading a tab-delimited data file in Windows with Pandas/Python without any problems. The data file contains notes in first three lines and then follows with a header.
```
df = pd.read_csv(myfile,sep='\t',skiprows=(0,1,2),header=(0))
```
I'm now trying to read this file with my Mac. (My first time using Python on Mac.) I get the following error.
```
pandas.parser.CParserError: Error tokenizing data. C error: Expected 1
fields in line 8, saw 39
```
If set the error_bad_lines argument for read_csv to False, I get the following information, which continues until the end of the last row.
```
Skipping line 8: expected 1 fields, saw 39
Skipping line 9: expected 1 fields, saw 125
Skipping line 10: expected 1 fields, saw 125
Skipping line 11: expected 1 fields, saw 125
Skipping line 12: expected 1 fields, saw 125
Skipping line 13: expected 1 fields, saw 125
Skipping line 14: expected 1 fields, saw 125
Skipping line 15: expected 1 fields, saw 125
Skipping line 16: expected 1 fields, saw 125
Skipping line 17: expected 1 fields, saw 125
...
```
Do I need to specify a value for the encoding argument? It seems as though I shouldn't have to because reading the file works fine on Windows.

Response:
The biggest clue is the rows are all being returned on one line. This indicates line terminators are being ignored or are not present. You can specify the line terminator for csv_reader. If you are on a mac the lines created will end with \rrather than the linux standard \n or better still the suspenders and belt approach of windows with \r\n.
```
pandas.read_csv(filename, sep='\t', lineterminator='\r')
```
You could also open all your data using the codecs package. This may increase robustness at the expense of document loading speed.
```
import codecs

doc = codecs.open('document','rU','UTF-16') #open for reading with "universal" type set

df = pandas.read_csv(doc, sep='\t')
```


Question: Pandas split DataFrame by column value I have DataFrame with column Sales. How can I split it into 2 based on Sales value? First DataFrame will have data with 'Sales' = s

Response:
You can use boolean indexing:
```
df = pd.DataFrame({'Sales':[10,20,30,40,50], 'A':[3,4,7,6,1]})
print (df)
   A  Sales
0  3     10
1  4     20
2  7     30
3  6     40
4  1     50

s = 30

df1 = df[df['Sales'] >= s]
print (df1)
   A  Sales
2  7     30
3  6     40
4  1     50

df2 = df[df['Sales'] < s]
print (df2)
   A  Sales
0  3     10
1  4     20
```
It's also possible to invert mask by ~:
```
mask = df['Sales'] >= s
df1 = df[mask]
df2 = df[~mask]
print (df1)
   A  Sales
2  7     30
3  6     40
4  1     50

print (df2)
   A  Sales
0  3     10
1  4     20
```
```
print (mask)
0    False
1    False
2     True
3     True
4     True
Name: Sales, dtype: bool

print (~mask)
0     True
1     True
2    False
3    False
4    False
Name: Sales, dtype: bool
```


Question:
How to read file with space separated values in pandas I try to read the file into pandas. The file has values separated by space, but with different number of spaces I tried:
```
pd.read_csv('file.csv', delimiter=' ')
```
but it doesn't work

Response: Add delim_whitespace=True argument, it's faster than regex. Note however that after pandas 2.2.0 this is deprecated and it's better to use sep="\s+"


Question:
Prevent pandas from interpreting 'NA' as NaN in a string The pandas read_csv() method interprets 'NA' as nan (not a number) instead of a valid string. In the simple case below note that the output in row 1, column 2 (zero based count) is 'nan' instead of 'NA'. sample.tsv (tab delimited) PDB CHAIN SP_PRIMARY RES_BEG RES_END PDB_BEG PDB_END SP_BEG SP_END 5d8b N P60490 1 146 1 146 1 146 5d8b NA P80377 1 126 1 126 1 126 5d8b O P60491 1 118 1 118 1 118 read_sample.py
```py
import pandas as pd

df = pd.read_csv(
    'sample.tsv',
    sep='\t',
    encoding='utf-8',
)

for df_tuples in df.itertuples(index=True):
    print(df_tuples)
```
output (0, u'5d8b', u'N', u'P60490', 1, 146, 1, 146, 1, 146) (1, u'5d8b', nan, u'P80377', 1, 126, 1, 126, 1, 126) (2, u'5d8b', u'O', u'P60491', 1, 118, 1, 118, 1, 118) Additional Information Re-writing the file with quotes for data in the 'CHAIN' column and then using the quotechar parameter quotechar='\'' has the same result. And passing a dictionary of types via the dtype parameter dtype=dict(valid_cols) does not change the result. An old answer to Prevent pandas from automatically inferring type in read_csv suggests first using a numpy record array to parse the file, but given the ability to now specify column dtypes, this shouldn't be necessary. Note that itertuples() is used to preserve dtypes as described in the iterrows documentation: "To preserve dtypes while iterating over the rows, it is better to use itertuples() which returns tuples of the values and which is generally faster as iterrows." Example was tested on Python 2 and 3 with pandas version 0.16.2, 0.17.0, and 0.17.1. Is there a way to capture a valid string 'NA' instead of it being converted to nan?

Response:
You could use parameters keep_default_na and na_values to set all NA values by hand docs:
```
import pandas as pd
from io import StringIO

data = """
PDB CHAIN SP_PRIMARY RES_BEG RES_END PDB_BEG PDB_END SP_BEG SP_END
5d8b N P60490 1 146 1 146 1 146
5d8b NA P80377 _ 126 1 126 1 126
5d8b O P60491 1 118 1 118 1 118
"""

df = pd.read_csv(StringIO(data), sep=' ', keep_default_na=False, na_values=['_'])

```python
df
#Output
#PDB CHAIN SP_PRIMARY  RES_BEG  RES_END  PDB_BEG  PDB_END  SP_BEG  SP_END
#0  5d8b     N     P60490        1      146        1      146       1     146
#1  5d8b    NA     P80377      NaN      126        1      126       1     126
#2  5d8b     O     P60491        1      118        1      118       1     118
```

```python
df.CHAIN.apply(type)
#Output
#0    <class 'str'>
#1    <class 'str'>
#2    <class 'str'>
#Name: CHAIN, dtype: object
#```
#EDIT All default NA values from na-values (as of pandas 1.0.0): The default NaN recognized values are ['-1.#IND', '1.#QNAN', '1.#IND', '-1.#QNAN', '#N/A N/A', '#N/A', 'N/A', 'n/a', 'NA', '', '#NA', 'NULL', 'null', 'NaN', '-NaN', 'nan', '-nan', ''].
```


Question:
Jupyter notebook display two pandas tables side by side I have two pandas dataframes and I would like to display them in Jupyter notebook. Doing something like:
```
display(df1)
display(df2)
```
Shows them one below another: I would like to have a second dataframe on the right of the first one. There is a similar question, but it looks like there a person is satisfied either with merging them in one dataframe of showing the difference between them. This will not work for me. In my case dataframes can represent completely different (non-comparable elements) and the size of them can be different. Thus my main goal is to save space.

Response:
I have ended up writing a function that can do this: [update: added titles based on suggestions (thnx @Antony_Hatchkins et al.)]
```
from IPython.display import display_html
from itertools import chain,cycle
def display_side_by_side(*args,titles=cycle([''])):
    html_str=''
    for df,title in zip(args, chain(titles,cycle(['</br>'])) ):
        html_str+='<th style="text-align:center"><td style="vertical-align:top">'
        html_str+=f'<h2 style="text-align: center;">{title}</h2>'
        html_str+=df.to_html().replace('table','table style="display:inline"')
        html_str+='</td></th>'
    display_html(html_str,raw=True)
```
Example usage:
```
df1 = pd.DataFrame(np.arange(12).reshape((3,4)),columns=['A','B','C','D',])
df2 = pd.DataFrame(np.arange(16).reshape((4,4)),columns=['A','B','C','D',])
display_side_by_side(df1,df2,df1, titles=['Foo','Foo Bar']) #we left 3rd empty...
```


Question:
Pandas aggregate count distinct Let's say I have a log of user activity and I want to generate a report of the total duration and the number of unique users per day.
```
import numpy as np
import pandas as pd
df = pd.DataFrame({'date': ['2013-04-01','2013-04-01','2013-04-01','2013-04-02', '2013-04-02'],
    'user_id': ['0001', '0001', '0002', '0002', '0002'],
    'duration': [30, 15, 20, 15, 30]})
```
Aggregating duration is pretty straightforward:
```
group = df.groupby('date')
agg = group.aggregate({'duration': np.sum})
agg
            duration
date
2013-04-01        65
2013-04-02        45
```
What I'd like to do is sum the duration and count distincts at the same time, but I can't seem to find an equivalent for count_distinct:
```
agg = group.aggregate({ 'duration': np.sum, 'user_id': count_distinct})
```
This works, but surely there's a better way, no?
```
group = df.groupby('date')
agg = group.aggregate({'duration': np.sum})
agg['uv'] = df.groupby('date').user_id.nunique()
agg
            duration  uv
date
2013-04-01        65   2
2013-04-02        45   1
```
I'm thinking I just need to provide a function that returns the count of distinct items of a Series object to the aggregate function, but I don't have a lot of exposure to the various libraries at my disposal. Also, it seems that the groupby object already knows this information, so wouldn't I just be duplicating the effort?

Response:
How about either of:
```

```python
df
#Output
#         date  duration user_id
#0  2013-04-01        30    0001
#1  2013-04-01        15    0001
#2  2013-04-01        20    0002
#3  2013-04-02        15    0002
#4  2013-04-02        30    0002
```

```python
df.groupby("date").agg({"duration": np.sum, "user_id": pd.Series.nunique})
#Output
#            duration  user_id
#date
#2013-04-01        65        2
#2013-04-02        45        1
```

```python
df.groupby("date").agg({"duration": np.sum, "user_id": lambda x: x.nunique()})
#Output
#            duration  user_id
#date
#2013-04-01        65        2
#2013-04-02        45        1
#```
```


Question: How to convert Pandas Series of dates string into date objects? I have a Pandas data frame and one of the columns, have dates in string format of YYYY-MM-DD. For e.g. : '2013-10-28' At the moment the dtype of the column is object. How do I convert the column values to Pandas date format?

Response:
Essentially equivalent to @waitingkuo, but I would use pd.to_datetime here (it seems a little cleaner, and offers some additional functionality e.g. dayfirst):
```

```python
df
#Output
#a        time
#0  1  2013-01-01
#1  2  2013-01-02
#2  3  2013-01-03
```

```python
pd.to_datetime(df['time'])
#Output
#0   2013-01-01 00:00:00
#1   2013-01-02 00:00:00
#2   2013-01-03 00:00:00
#Name: time, dtype: datetime64[ns]
```

```python
df['time'] = pd.to_datetime(df['time'])
```

```python
df
#Output
#a                time
#0  1 2013-01-01 00:00:00
#1  2 2013-01-02 00:00:00
#2  3 2013-01-03 00:00:00
#```
#Handling ValueErrors If you run into a situation where doing
#```
#df['time'] = pd.to_datetime(df['time'])
#```
#Throws a
#```
#ValueError: Unknown string format
#```
#That means you have invalid (non-coercible) values. If you are okay with having them converted to pd.NaT, you can add an errors='coerce' argument to to_datetime:
#```
#df['time'] = pd.to_datetime(df['time'], errors='coerce')
#```
```


Question:
Get a frequency count based on multiple dataframe columns I have the following dataframe. Group Size Short Small Short Small Moderate Medium Moderate Small Tall Large I want to count the frequency of how many times the same row appears in the dataframe.
```
Group           Size      Time
Short          Small        2
Moderate       Medium       1
Moderate       Small        1
Tall           Large        1
```

Response:
You can use groupby's size
```py
import pandas as pd

# load the sample data
data = {'Group': ['Short', 'Short', 'Moderate', 'Moderate', 'Tall'], 'Size': ['Small', 'Small', 'Medium', 'Small', 'Large']}
df = pd.DataFrame(data)
```
Option 1:
```py
dfg = df.groupby(by=["Group", "Size"]).size()

# which results in a pandas.core.series.Series
Group     Size
Moderate  Medium    1
          Small     1
Short     Small     2
Tall      Large     1
dtype: int64
```
Option 2:
```py
dfg = df.groupby(by=["Group", "Size"]).size().reset_index(name="Time")

# which results in a pandas.core.frame.DataFrame
      Group    Size  Time
0  Moderate  Medium     1
1  Moderate   Small     1
2     Short   Small     2
3      Tall   Large     1
```
Option 3:
```py
dfg = df.groupby(by=["Group", "Size"], as_index=False).size()

# which results in a pandas.core.frame.DataFrame
      Group    Size  Time
0  Moderate  Medium     1
1  Moderate   Small     1
2     Short   Small     2
3      Tall   Large     1
```


Question:
What are the pros and cons between get_dummies (Pandas) and OneHotEncoder (Scikit-learn)? I'm learning different methods to convert categorical variables to numeric for machine-learning classifiers. I came across the pd.get_dummies method and sklearn.preprocessing.OneHotEncoder() and I wanted to see how they differed in terms of performance and usage. I found a tutorial on how to use OneHotEncoder() on https://xgdgsc.wordpress.com/2015/03/20/note-on-using-onehotencoder-in-scikit-learn-to-work-on-categorical-features/ since the sklearn documentation wasn't too helpful on this feature. I have a feeling I'm not doing it correctly...but Can some explain the pros and cons of using pd.dummies over sklearn.preprocessing.OneHotEncoder() and vice versa? I know that OneHotEncoder() gives you a sparse matrix but other than that I'm not sure how it is used and what the benefits are over the pandas method. Am I using it inefficiently?
```
import pandas as pd
import numpy as np
from sklearn.datasets import load_iris
sns.set()

%matplotlib inline

#Iris Plot
iris = load_iris()
n_samples, m_features = iris.data.shape

#Load Data
X, y = iris.data, iris.target
D_target_dummy = dict(zip(np.arange(iris.target_names.shape[0]), iris.target_names))

DF_data = pd.DataFrame(X,columns=iris.feature_names)
DF_data["target"] = pd.Series(y).map(D_target_dummy)
#sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)  \
#0                  5.1               3.5                1.4               0.2
#1                  4.9               3.0                1.4               0.2
#2                  4.7               3.2                1.3               0.2
#3                  4.6               3.1                1.5               0.2
#4                  5.0               3.6                1.4               0.2
#5                  5.4               3.9                1.7               0.4

DF_dummies = pd.get_dummies(DF_data["target"])
#setosa  versicolor  virginica
#0         1           0          0
#1         1           0          0
#2         1           0          0
#3         1           0          0
#4         1           0          0
#5         1           0          0

from sklearn.preprocessing import OneHotEncoder, LabelEncoder
def f1(DF_data):
    Enc_ohe, Enc_label = OneHotEncoder(), LabelEncoder()
    DF_data["Dummies"] = Enc_label.fit_transform(DF_data["target"])
    DF_dummies2 = pd.DataFrame(Enc_ohe.fit_transform(DF_data[["Dummies"]]).todense(), columns = Enc_label.classes_)
    return(DF_dummies2)

%timeit pd.get_dummies(DF_data["target"])
#1000 loops, best of 3: 777 µs per loop

%timeit f1(DF_data)
#100 loops, best of 3: 2.91 ms per loop
```

Response:
For machine learning, you almost definitely want to use sklearn.OneHotEncoder. For other tasks like simple analyses, you might be able to use pd.get_dummies, which is a bit more convenient. Note that sklearn.OneHotEncoder has been updated in the latest version so that it does accept strings for categorical variables, as well as integers. The crux of it is that the sklearn encoder creates a function which persists and can then be applied to new data sets which use the same categorical variables, with consistent results.
```
from sklearn.preprocessing import OneHotEncoder

# Create the encoder.
encoder = OneHotEncoder(handle_unknown="ignore")
encoder.fit(X_train)    # Assume for simplicity all features are categorical.

# Apply the encoder.
X_train = encoder.transform(X_train)
X_test = encoder.transform(X_test)
```
Note how we apply the same encoder we created via X_train to the new data set X_test. Consider what happens if X_test contains different levels than X_train for one of its variables. For example, let's say X_train["color"] contains only "red" and "green", but in addition to those, X_test["color"] sometimes contains "blue". If we use pd.get_dummies, X_test will end up with an additional "color_blue" column which X_train doesn't have, and the inconsistency will probably break our code later on, especially if we are feeding X_test to an sklearn model which we trained on X_train. And if we want to process the data like this in production, where we're receiving a single example at a time, pd.get_dummies won't be of use. With sklearn.OneHotEncoder on the other hand, once we've created the encoder, we can reuse it to produce the same output every time, with columns only for "red" and "green". And we can explicitly control what happens when it encounters the new level "blue": if we think that's impossible, then we can tell it to throw an error with handle_unknown="error"; otherwise we can tell it to continue and simply set the red and green columns to 0, with handle_unknown="ignore".


Question:
Appending pandas dataframes generated in a for loop I am accessing a series of Excel files in a for loop. I then read the data in the excel file to a pandas dataframe. I cant figure out how to append these dataframes together to then save the dataframe (now containing the data from all the files) as a new Excel file. Here's what I tried:
```
for infile in glob.glob("*.xlsx"):
    data = pandas.read_excel(infile)
    appended_data = pandas.DataFrame.append(data) # requires at least two arguments
appended_data.to_excel("appended.xlsx")
```
Thanks!

Response:
Use pd.concat to merge a list of DataFrame into a single big DataFrame.
```
appended_data = []
for infile in glob.glob("*.xlsx"):
    data = pandas.read_excel(infile)
    # store DataFrame in list
    appended_data.append(data)
# see pd.concat documentation for more info
appended_data = pd.concat(appended_data)
# write DataFrame to an excel sheet
appended_data.to_excel('appended.xlsx')
```


Question:
Pandas Left Outer Join results in table larger than left table From what I understand about a left outer join, the resulting table should never have more rows than the left table...Please let me know if this is wrong... My left table is 192572 rows and 8 columns. My right table is 42160 rows and 5 columns. My Left table has a field called 'id' which matches with a column in my right table called 'key'. Therefore I merge them as such:
```
combined = pd.merge(a,b,how='left',left_on='id',right_on='key')
```
But then the combined shape is 236569. What am I misunderstanding?

Response:
You can expect this to increase if keys match more than one row in the other DataFrame:
```

```python
df = pd.DataFrame([[1, 3], [2, 4]], columns=['A', 'B'])
```

```python
df2 = pd.DataFrame([[1, 5], [1, 6]], columns=['A', 'C'])
```

```python
df.merge(df2, how='left')  # merges on columns A
#Output
#A  B   C
#0  1  3   5
#1  1  3   6
#2  2  4 NaN
#```
#To avoid this behaviour drop the duplicates in df2:
#```
```

```python
df2.drop_duplicates(subset=['A'])  # you can use take_last=True
#Output
#A  C
#0  1  5
```

```python
df.merge(df2.drop_duplicates(subset=['A']), how='left')
#Output
#A  B   C
#0  1  3   5
#1  2  4 NaN
#```
```


Question:
Efficiently checking if arbitrary object is NaN in Python / numpy / pandas? My numpy arrays use np.nan to designate missing values. As I iterate over the data set, I need to detect such missing values and handle them in special ways. Naively I used numpy.isnan(val), which works well unless val isn't among the subset of types supported by numpy.isnan(). For example, missing data can occur in string fields, in which case I get:
```

```python
np.isnan('some_string')
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#TypeError: Not implemented for this type
#```
#Other than writing an expensive wrapper that catches the exception and returns False, is there a way to handle this elegantly and efficiently?
```

Response:
pandas.isnull() (also pd.isna(), in newer versions) checks for missing values in both numeric and string/object arrays. From the documentation, it checks for: NaN in numeric arrays, None/NaN in object arrays Quick example:
```
import pandas as pd
import numpy as np
s = pd.Series(['apple', np.nan, 'banana'])
pd.isnull(s)
Out[9]:
0    False
1     True
2    False
dtype: bool
```
The idea of using numpy.nan to represent missing values is something that pandas introduced, which is why pandas has the tools to deal with it. Datetimes too (if you use pd.NaT you won't need to specify the dtype)
```

```python
s = Series([Timestamp('20130101'),np.nan,Timestamp('20130102 9:30')],dtype='M8[ns]')
```

```python
s
#Output
#0   2013-01-01 00:00:00
#1                   NaT
#2   2013-01-02 09:30:00
#dtype: datetime64[ns]``
```

```python
pd.isnull(s)
#Output
#0    False
#1     True
#2    False
#dtype: bool
#```
```


Question:
Count unique values using pandas groupby [duplicate] This question already has answers here: Pandas 'count(distinct)' equivalent (12 answers) Closed 2 years ago. I have data of the following form:
```
df = pd.DataFrame({
    'group': [1, 1, 2, 3, 3, 3, 4],
    'param': ['a', 'a', 'b', np.nan, 'a', 'a', np.nan]
})
print(df)

#    group param
# 0      1     a
# 1      1     a
# 2      2     b
# 3      3   NaN
# 4      3     a
# 5      3     a
# 6      4   NaN
```
Non-null values within groups are always the same. I want to count the non-null value for each group (where it exists) once, and then find the total counts for each value. I'm currently doing this in the following (clunky and inefficient) way:
```
param = []
for _, group in df[df.param.notnull()].groupby('group'):
    param.append(group.param.unique()[0])
print(pd.DataFrame({'param': param}).param.value_counts())

# a    2
# b    1
```
I'm sure there's a way to do this more cleanly and without using a loop, but I just can't seem to work it out. Any help would be much appreciated.

Response:
I think you can use SeriesGroupBy.nunique:
```
print (df.groupby('param')['group'].nunique())
param
a    2
b    1
Name: group, dtype: int64
```
Another solution with unique, then create new df by DataFrame.from_records, reshape to Series by stack and last value_counts:
```
a = df[df.param.notnull()].groupby('group')['param'].unique()
print (pd.DataFrame.from_records(a.values.tolist()).stack().value_counts())
a    2
b    1
dtype: int64
```


Question: Skip rows during csv import pandas I'm trying to import a .csv file using pandas.read_csv(), however, I don't want to import the 2nd row of the data file (the row with index = 1 for 0-indexing). I can't see how not to import it because the arguments used with the command seem ambiguous: From the pandas website: skiprows : list-like or integer Row numbers to skip (0-indexed) or number of rows to skip (int) at the start of the file." If I put skiprows=1 in the arguments, how does it know whether to skip the first row or skip the row with index 1?

Response:
You can try yourself:
```

```python
import pandas as pd
```

```python
from io import StringIO
```

```python
s = """1, 2
3, 4
5, 6"""
```

```python
pd.read_csv(StringIO(s), skiprows=[1], header=None)
#Output
#   0  1
#0  1  2
#1  5  6
```

```python
pd.read_csv(StringIO(s), skiprows=1, header=None)
#Output
#   0  1
#0  3  4
#1  5  6
#```
```


Question:
How to filter in NaN (pandas)? I have a pandas dataframe (df), and I want to do something like:
```
newdf = df[(df.var1 == 'a') & (df.var2 == NaN)]
```
I've tried replacing NaN with np.NaN, or 'NaN' or 'nan' etc, but nothing evaluates to True. There's no pd.NaN. I can use df.fillna(np.nan) before evaluating the above expression but that feels hackish and I wonder if it will interfere with other pandas operations that rely on being able to identify pandas-format NaN's later. I get the feeling there should be an easy answer to this question, but somehow it has eluded me.

Response:
```
filtered_df = df[df['var2'].isna()]
```
This filters and gives you rows which has only NaN values in 'var2' column. Note: "Series.isnull is an alias for Series.isna."


Question:
Display rows with one or more NaN values in pandas dataframe I have a dataframe in which some rows contain missing values.
```

```python
df.head()
#Output
#alpha1  alpha2    gamma1    gamma2       chi2min
#filename
#M66_MI_NSRh35d32kpoints.dat  0.8016  0.9283  1.000000  0.074804  3.985599e+01
#F71_sMI_DMRI51d.dat          0.0000  0.0000       NaN  0.000000  1.000000e+25
#F62_sMI_St22d7.dat           1.7210  3.8330  0.237480  0.150000  1.091832e+01
#F41_Car_HOC498d.dat          1.1670  2.8090  0.364190  0.300000  7.966335e+00
#F78_MI_547d.dat              1.8970  5.4590  0.095319  0.100000  2.593468e+01
#```
#I want to display those rows on the screen. If I try df.isnull(), it gives a long dataframe with True and False. Is there any way by which I can select these rows and print them on the screen?
```

Response:
You can use DataFrame.any with parameter axis=1 for check at least one True in row by DataFrame.isna with boolean indexing:
```
df1 = df[df.isna().any(axis=1)]
```
```
d = {'filename': ['M66_MI_NSRh35d32kpoints.dat', 'F71_sMI_DMRI51d.dat', 'F62_sMI_St22d7.dat', 'F41_Car_HOC498d.dat', 'F78_MI_547d.dat'], 'alpha1': [0.8016, 0.0, 1.721, 1.167, 1.897], 'alpha2': [0.9283, 0.0, 3.833, 2.809, 5.459], 'gamma1': [1.0, np.nan, 0.23748000000000002, 0.36419, 0.095319], 'gamma2': [0.074804, 0.0, 0.15, 0.3, np.nan], 'chi2min': [39.855990000000006, 1e+25, 10.91832, 7.966335000000001, 25.93468]}
df = pd.DataFrame(d).set_index('filename')
```
```
print (df)
                             alpha1  alpha2    gamma1    gamma2       chi2min
filename
M66_MI_NSRh35d32kpoints.dat  0.8016  0.9283  1.000000  0.074804  3.985599e+01
F71_sMI_DMRI51d.dat          0.0000  0.0000       NaN  0.000000  1.000000e+25
F62_sMI_St22d7.dat           1.7210  3.8330  0.237480  0.150000  1.091832e+01
F41_Car_HOC498d.dat          1.1670  2.8090  0.364190  0.300000  7.966335e+00
F78_MI_547d.dat              1.8970  5.4590  0.095319       NaN  2.593468e+01
```
Explanation:
```
print (df.isna())
                            alpha1 alpha2 gamma1 gamma2 chi2min
filename
M66_MI_NSRh35d32kpoints.dat  False  False  False  False   False
F71_sMI_DMRI51d.dat          False  False   True  False   False
F62_sMI_St22d7.dat           False  False  False  False   False
F41_Car_HOC498d.dat          False  False  False  False   False
F78_MI_547d.dat              False  False  False   True   False

print (df.isna().any(axis=1))
filename
M66_MI_NSRh35d32kpoints.dat    False
F71_sMI_DMRI51d.dat             True
F62_sMI_St22d7.dat             False
F41_Car_HOC498d.dat            False
F78_MI_547d.dat                 True
dtype: bool

df1 = df[df.isna().any(axis=1)]
print (df1)
                     alpha1  alpha2    gamma1  gamma2       chi2min
filename
F71_sMI_DMRI51d.dat   0.000   0.000       NaN     0.0  1.000000e+25
F78_MI_547d.dat       1.897   5.459  0.095319     NaN  2.593468e+01
```


Question:
Normalize data in pandas Suppose I have a pandas data frame df: I want to calculate the column wise mean of a data frame. This is easy:
```
df.apply(average)
```
then the column wise range max(col) - min(col). This is easy again:
```
df.apply(max) - df.apply(min)
```
Now for each element I want to subtract its column's mean and divide by its column's range. I am not sure how to do that Any help/pointers are much appreciated.

Response:
```

```python
df
#Output
#a         b          c         d
#A  -0.488816  0.863769   4.325608 -4.721202
#B -11.937097  2.993993 -12.916784 -1.086236
#C  -5.569493  4.672679  -2.168464 -9.315900
#D   8.892368  0.932785   4.535396  0.598124
```

```python
df_norm = (df - df.mean()) / (df.max() - df.min())
```

```python
df_norm
#Output
#a         b         c         d
#A  0.085789 -0.394348  0.337016 -0.109935
#B -0.463830  0.164926 -0.650963  0.256714
#C -0.158129  0.605652 -0.035090 -0.573389
#D  0.536170 -0.376229  0.349037  0.426611
```

```python
df_norm.mean()
#Output
#a   -2.081668e-17
#b    4.857226e-17
#c    1.734723e-17
#d   -1.040834e-17
```

```python
df_norm.max() - df_norm.min()
#Output
#a    1
#b    1
#c    1
#d    1
#```
```


Question:
Can Pandas plot a histogram of dates? I've taken my Series and coerced it to a datetime column of dtype=datetime64[ns] (though only need day resolution...not sure how to change).
```
import pandas as pd
df = pd.read_csv('somefile.csv')
column = df['date']
column = pd.to_datetime(column, coerce=True)
```
but plotting doesn't work:
```
ipdb> column.plot(kind='hist')
*** TypeError: ufunc add cannot use operands with types dtype('<M8[ns]') and dtype('float64')
```
I'd like to plot a histogram that just shows the count of dates by week, month, or year. Surely there is a way to do this in pandas?

Response:
Given this df:
```
date
0 2001-08-10
1 2002-08-31
2 2003-08-29
3 2006-06-21
4 2002-03-27
5 2003-07-14
6 2004-06-15
7 2003-08-14
8 2003-07-29
```
and, if it's not already the case:
```
df["date"] = df["date"].astype("datetime64")
```
To show the count of dates by month:
```
df.groupby(df["date"].dt.month).count().plot(kind="bar")
```
.dt allows you to access the datetime properties. Which will give you: You can replace month by year, day, etc.. If you want to distinguish year and month for instance, just do:
```
df.groupby([df["date"].dt.year, df["date"].dt.month]).count().plot(kind="bar")
```
Which gives:


Question:
How do I delete a column that contains only zeros in Pandas? I currently have a dataframe consisting of columns with 1's and 0's as values, I would like to iterate through the columns and delete the ones that are made up of only 0's. Here's what I have tried so far:
```
ones = []
zeros = []
for year in years:
    for i in range(0,599):
        if year[str(i)].values.any() == 1:
            ones.append(i)
        if year[str(i)].values.all() == 0:
            zeros.append(i)
    for j in ones:
        if j in zeros:
            zeros.remove(j)
    for q in zeros:
        del year[str(q)]
```
In which years is a list of dataframes for the various years I am analyzing, ones consists of columns with a one in them and zeros is a list of columns containing all zeros. Is there a better way to delete a column based on a condition? For some reason I have to check whether the ones columns are in the zeros list as well and remove them from the zeros list to obtain a list of all the zero columns.

Response:
```
df.loc[:, (df != 0).any(axis=0)]
```
Here is a break-down of how it works:
```

```python
import pandas as pd
```

```python
df = pd.DataFrame([[1,0,0,0], [0,0,1,0]])
```

```python
df
#Output
#0  1  2  3
#0  1  0  0  0
#1  0  0  1  0
#
#[2 rows x 4 columns]
#```
#df != 0 creates a boolean DataFrame which is True where df is nonzero:
#```
```

```python
df != 0
#Output
#0      1      2      3
#0   True  False  False  False
#1  False  False   True  False
#
#[2 rows x 4 columns]
#```
#(df != 0).any(axis=0) returns a boolean Series indicating which columns have nonzero entries. (The any operation aggregates values along the 0-axis -- i.e. along the rows -- into a single boolean value. Hence the result is one boolean value for each column.)
#```
```

```python
(df != 0).any(axis=0)
#Output
#0     True
#1    False
#2     True
#3    False
#dtype: bool
#```
#And df.loc can be used to select those columns:
#```
```

```python
df.loc[:, (df != 0).any(axis=0)]
#Output
#0  2
#0  1  0
#1  0  1
#
#[2 rows x 2 columns]
#```
#To "delete" the zero-columns, reassign df:
#```
#df = df.loc[:, (df != 0).any(axis=0)]
#```
```


Question:
Pandas plot doesn't show When using this in a script (not IPython), nothing happens, i.e. the plot window doesn't appear :
```
import numpy as np
import pandas as pd
ts = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', periods=1000))
ts.plot()
```
Even when adding time.sleep(5), there is still nothing. Why? Is there a way to do it, without having to manually call matplotlib ?

Response:
Once you have made your plot, you need to tell matplotlib to show it. The usual way to do things is to import matplotlib.pyplot and call show from there:
```
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
ts = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', periods=1000))
ts.plot()
plt.show()
```
In older versions of pandas, you were able to find a backdoor to matplotlib, as in the example below. NOTE: This no longer works in modern versions of pandas, and I still recommend importing matplotlib separately, as in the example above.
```
import numpy as np
import pandas as pd
ts = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', periods=1000))
ts.plot()
pd.tseries.plotting.pylab.show()
```
But all you are doing there is finding somewhere that matplotlib has been imported in pandas, and calling the same show function from there. Are you trying to avoid calling matplotlib in an effort to speed things up? If so then you are really not speeding anything up, since pandas already imports pyplot:
```
python -mtimeit -s 'import pandas as pd'
100000000 loops, best of 3: 0.0122 usec per loop

python -mtimeit -s 'import pandas as pd; import matplotlib.pyplot as plt'
100000000 loops, best of 3: 0.0125 usec per loop
```
Finally, the reason the example you linked in comments doesn't need the call to matplotlib is because it is being run interactively in an iPython notebook, not in a script.


Question: How to replace negative numbers in Pandas Data Frame by zero I would like to know if there is someway of replacing all DataFrame negative numbers by zeros?

Response:
If all your columns are numeric, you can use boolean indexing:
```

```python
import pandas as pd
```

```python
df = pd.DataFrame({'a': [0, -1, 2], 'b': [-3, 2, 1]})
```

```python
df
#Output
#a  b
#0  0 -3
#1 -1  2
#2  2  1
```

```python
df[df < 0] = 0
```

```python
df
#Output
#a  b
#0  0  0
#1  0  2
#2  2  1
#```
#For the more general case, this answer shows the private method _get_numeric_data:
#```
```

```python
import pandas as pd
```

```python
df = pd.DataFrame({'a': [0, -1, 2], 'b': [-3, 2, 1],
                           'c': ['foo', 'goo', 'bar']})
```

```python
df
#Output
#a  b    c
#0  0 -3  foo
#1 -1  2  goo
#2  2  1  bar
```

```python
num = df._get_numeric_data()
```

```python
num[num < 0] = 0
```

```python
df
#Output
#a  b    c
#0  0  0  foo
#1  0  2  goo
#2  2  1  bar
#```
#With timedelta type, boolean indexing seems to work on separate columns, but not on the whole dataframe. So you can do:
#```
```

```python
import pandas as pd
```

```python
df = pd.DataFrame({'a': pd.to_timedelta([0, -1, 2], 'd'),
   ...:                    'b': pd.to_timedelta([-3, 2, 1], 'd')})
```

```python
df
#Output
#a       b
#0  0 days -3 days
#1 -1 days  2 days
#2  2 days  1 days
```

```python
for k, v in df.iteritems():
   ...:     v[v < 0] = 0
   ...:
```

```python
df
#Output
#a      b
#0 0 days 0 days
#1 0 days 2 days
#2 2 days 1 days
#```
#Update: comparison with a pd.Timedelta works on the whole DataFrame:
#```
```

```python
import pandas as pd
```

```python
df = pd.DataFrame({'a': pd.to_timedelta([0, -1, 2], 'd'),
   ...:                    'b': pd.to_timedelta([-3, 2, 1], 'd')})
```

```python
df[df < pd.Timedelta(0)] = 0
```

```python
df
#Output
#a      b
#0 0 days 0 days
#1 0 days 2 days
#2 2 days 1 days
#```
```


Question:
Pandas groupby: How to get a union of strings I have a dataframe like this:
```
A         B       C
0  1  0.749065    This
1  2  0.301084      is
2  3  0.463468       a
3  4  0.643961  random
4  1  0.866521  string
5  2  0.120737       !
```
Calling
```

```python
print df.groupby("A")["B"].sum()
```
will return
```
A
1    1.615586
2    0.421821
3    0.463468
4    0.643961
```
Now I would like to do "the same" for column "C". Because that column contains strings, sum() doesn't work (although you might think that it would concatenate the strings). What I would really like to see is a list or set of the strings for each group, i.e.
```
A
1    {This, string}
2    {is, !}
3    {a}
4    {random}
```
I have been trying to find ways to do this. Series.unique() (http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.unique.html) doesn't work, although
```
df.groupby("A")["B"]
```
is a
```
pandas.core.groupby.SeriesGroupBy object
```
so I was hoping any Series method would work. Any ideas?
```

Response:
```

```python
df = read_csv(StringIO(data),sep='\s+')
```

```python
df
#Output
#A         B       C
#0  1  0.749065    This
#1  2  0.301084      is
#2  3  0.463468       a
#3  4  0.643961  random
#4  1  0.866521  string
#5  2  0.120737       !
```

```python
df.dtypes
#Output
#A      int64
#B    float64
#C     object
#dtype: object
#```
#When you apply your own function, there is not automatic exclusions of non-numeric columns. This is slower, though, than the application of .sum() to the groupby
#```
```

```python
df.groupby('A').apply(lambda x: x.sum())
#Output
#A         B           C
#A
#1  2  1.615586  Thisstring
#2  4  0.421821         is!
#3  3  0.463468           a
#4  4  0.643961      random
#```
#sum by default concatenates
#```
```

```python
df.groupby('A')['C'].apply(lambda x: x.sum())
#Output
#A
#1    Thisstring
#2           is!
#3             a
#4        random
#dtype: object
#```
#You can do pretty much what you want
#```
```

```python
df.groupby('A')['C'].apply(lambda x: "{%s}" % ', '.join(x))
#Output
#A
#1    {This, string}
#2           {is, !}
#3               {a}
#4          {random}
#dtype: object
#```
#Doing this on a whole frame, one group at a time. Key is to return a Series
#```
#def f(x):
#     return Series(dict(A = x['A'].sum(),
#                        B = x['B'].sum(),
#                        C = "{%s}" % ', '.join(x['C'])))
```

```python
df.groupby('A').apply(f)
#Output
#A         B               C
#A
#1  2  1.615586  {This, string}
#2  4  0.421821         {is, !}
#3  3  0.463468             {a}
#4  4  0.643961        {random}
#```
```


Question:
How to set the xlim and ylim of a FacetGrid I'm using sns.lmplot to plot a linear regression, dividing my dataset into two groups with a categorical variable. For both x and y, I'd like to manually set the lower bound on both plots, but leave the upper bound at the Seaborn default. Here's a simple example:
```
import pandas as pd
import seaborn as sns
import numpy as np

n = 200
np.random.seed(2014)
base_x = np.random.rand(n)
base_y = base_x * 2
errors = np.random.uniform(size=n)
y = base_y + errors

df = pd.DataFrame({'X': base_x, 'Y': y, 'Z': ['A','B']*(100)})

mask_for_b = df.Z == 'B'
df.loc[mask_for_b,['X','Y']] = df.loc[mask_for_b,] *2

sns.lmplot('X','Y',df,col='Z',sharex=False,sharey=False)
```
This outputs the following: But in this example, I'd like the xlim and the ylim to be (0,*) . I tried using sns.plt.ylim and sns.plt.xlim but those only affect the right-hand plot. Example:
```
sns.plt.ylim(0,)
sns.plt.xlim(0,)
```
How can I access the xlim and ylim for each plot in the FacetGrid?

Response:
The lmplot function returns a FacetGrid instance. This object has a method called set, to which you can pass key=value pairs and they will be set on each Axes object in the grid. Secondly, you can set only one side of an Axes limit in matplotlib by passing None for the value you want to remain as the default. Putting these together, we have:
```
g = sns.lmplot('X', 'Y', df, col='Z', sharex=False, sharey=False)
g.set(ylim=(0, None))
```
Update Positional arguments, sharex and sharey are deprecate beginning in seaborn 0.11
```py
g = sns.lmplot(x='X', y='Y', data=df, col='Z', facet_kws={'sharey': False, 'sharex': False})
g.set(ylim=(0, None))
```


Question:
Get first element of Series without knowing the index [duplicate] This question already has answers here: Pandas: Get first row value of a given column (10 answers) Closed 7 years ago. Is there any way to access the first element of a Series without knowing its index? Let's say I have the following Series:
```
import pandas as pd

key='MCS096'
SUBJECTS = pd.DataFrame(
    {
        "ID": pd.Series([146], index=[145]),
        "study": pd.Series(["MCS"], index=[145]),
        "center": pd.Series(["Mag"], index=[145]),
        "initials": pd.Series(["MCS096"], index=[145]),
    }
)
```
Print out SUBJECTS:
```
print(SUBJECTS[SUBJECTS.initials==key]['ID'])

```python
145    146
```

```python
Name: ID, dtype: int64
#Output
#```
#How can I get the value 146 here without using index 145?
```

Response:
Use iloc to access by position (rather than label):
```

```python
df = pd.DataFrame([[1, 2], [3, 4]], ['a', 'b'], ['A', 'B'])
```

```python
df
#Output
#A  B
#a  1  2
#b  3  4
```

```python
df.iloc[0]  # first row in a DataFrame
#Output
#A    1
#B    2
#Name: a, dtype: int64
```

```python
df['A'].iloc[0]  # first item in a Series (Column)
#Output
#1
#```
```


Question:
Select latest in each group in a pandas dataframe How to group values of pandas dataframe and select the latest (by date) from each group? For example, given a dataframe sorted by date:
```
id     product   date
0   220    6647     2014-09-01
1   220    6647     2014-09-03
2   220    6647     2014-10-16
3   826    3380     2014-11-11
4   826    3380     2014-12-09
5   826    3380     2015-05-19
6   901    4555     2014-09-01
7   901    4555     2014-10-05
8   901    4555     2014-11-01
```
grouping by id or product, and selecting the latest gives:
```
id     product   date
2   220    6647     2014-10-16
5   826    3380     2015-05-19
8   901    4555     2014-11-01
```

Response:
You can also use tail with groupby to get the last n values of the group:
```
df.sort_values('date').groupby('id').tail(1)

    id  product date
2   220 6647    2014-10-16
8   901 4555    2014-11-01
5   826 3380    2015-05-19
```


Question: How can I filter lines on load in Pandas read_csv function? How can I filter which lines of a CSV to be loaded into memory using pandas? This seems like an option that one should find in read_csv. Am I missing something? Example: we've a CSV with a timestamp column and we'd like to load just the lines that with a timestamp greater than a given constant.

Response:
There isn't an option to filter the rows before the CSV file is loaded into a pandas object. You can either load the file and then filter using df[df['field'] > constant], or if you have a very large file and you are worried about memory running out, then use an iterator and apply the filter as you concatenate chunks of your file e.g.:
```
import pandas as pd
iter_csv = pd.read_csv('file.csv', iterator=True, chunksize=1000)
df = pd.concat([chunk[chunk['field'] > constant] for chunk in iter_csv])
```
You can vary the chunksize to suit your available memory. See here for more details.


Question:
Output data from all columns in a dataframe in pandas [duplicate] This question already has answers here: How do I expand the output display to see more columns of a Pandas DataFrame? (23 answers) Closed 6 years ago. I have a csv file with the name params.csv. I opened up ipython qtconsole and created a pandas dataframe using:
```
import pandas
paramdata = pandas.read_csv('params.csv', names=paramnames)
```
where, paramnames is a python list of string objects. Example of paramnames (the length of actual list is 22):
```
paramnames = ["id",
"fc",
"mc",
"markup",
"asplevel",
"aspreview",
"reviewpd"]
```
At the ipython prompt if I type paramdata and press enter then I do not get the dataframe with columns and values as shown in examples on Pandas website. Instead, I get information about the dataframe. I get:
```

```python
paramdata
#Output
#<class 'pandas.core.frame.DataFrame'>
#Int64Index: 59 entries, 0 to 58
#Data columns:
#id                    59  non-null values
#fc                    59  non-null values
#mc                    59  non-null values
#markup                59  non-null values
#asplevel              59  non-null values
#aspreview             59  non-null values
#reviewpd              59  non-null values
#```
#If I type paramdata['mc'] then I do get the values as expected for the mc column. I have two questions: (1) In the examples on the pandas website (see, for example, the output of df here: http://pandas.sourceforge.net/indexing.html#additional-column-access) typing the name of the dataframe gives the actual data. Why am I getting information about the dataframe as shown above instead of the actual data? Do I need to set some output options somewhere? (2) How do I output all columns in the dataframe to the screen without having to type their names, i.e., without having to type something like paramdata[['id','fc','mc']]. I am using pandas version 0.8. Thank you.
```

Response:
Use:
```
pandas.set_option('display.max_columns', 7)
```
This will force Pandas to display the 7 columns you have. Or more generally:
```
pandas.set_option('display.max_columns', None)
```
which will force it to display any number of columns. Explanation: the default for max_columns is 0, which tells Pandas to display the table only if all the columns can be squeezed into the width of your console. Alternatively, you can change the console width (in chars) from the default of 80 using e.g:
```
pandas.set_option('display.width', 200)
```


Question:
Read specific columns with pandas or other python module I have a csv file from this webpage. I want to read some of the columns in the downloaded file (the csv version can be downloaded in the upper right corner). Let's say I want 2 columns: 59 which in the header is star_name 60 which in the header is ra. However, for some reason the authors of the webpage sometimes decide to move the columns around. In the end I want something like this, keeping in mind that values can be missing.
```
data = #read data in a clever way
names = data['star_name']
ras = data['ra']
```
This will prevent my program to malfunction when the columns are changed again in the future, if they keep the name correct. Until now I have tried various ways using the csv module and resently the pandas module. Both without any luck. EDIT (added two lines + the header of my datafile. Sorry, but it's extremely long.)
```
# name, mass, mass_error_min, mass_error_max, radius, radius_error_min, radius_error_max, orbital_period, orbital_period_err_min, orbital_period_err_max, semi_major_axis, semi_major_axis_error_min, semi_major_axis_error_max, eccentricity, eccentricity_error_min, eccentricity_error_max, angular_distance, inclination, inclination_error_min, inclination_error_max, tzero_tr, tzero_tr_error_min, tzero_tr_error_max, tzero_tr_sec, tzero_tr_sec_error_min, tzero_tr_sec_error_max, lambda_angle, lambda_angle_error_min, lambda_angle_error_max, impact_parameter, impact_parameter_error_min, impact_parameter_error_max, tzero_vr, tzero_vr_error_min, tzero_vr_error_max, K, K_error_min, K_error_max, temp_calculated, temp_measured, hot_point_lon, albedo, albedo_error_min, albedo_error_max, log_g, publication_status, discovered, updated, omega, omega_error_min, omega_error_max, tperi, tperi_error_min, tperi_error_max, detection_type, mass_detection_type, radius_detection_type, alternate_names, molecules, star_name, ra, dec, mag_v, mag_i, mag_j, mag_h, mag_k, star_distance, star_metallicity, star_mass, star_radius, star_sp_type, star_age, star_teff, star_detected_disc, star_magnetic_field
11 Com b,19.4,1.5,1.5,,,,326.03,0.32,0.32,1.29,0.05,0.05,0.231,0.005,0.005,0.011664,,,,,,,,,,,,,,,,,,,,,,,,,,,,,1,2008,2011-12-23,94.8,1.5,1.5,2452899.6,1.6,1.6,Radial Velocity,,,,,11 Com,185.1791667,17.7927778,4.74,,,,,110.6,-0.35,2.7,19.0,G8 III,,4742.0,,
11 UMi b,10.5,2.47,2.47,,,,516.22,3.25,3.25,1.54,0.07,0.07,0.08,0.03,0.03,0.012887,,,,,,,,,,,,,,,,,,,,,,,,,,,,,1,2009,2009-08-13,117.63,21.06,21.06,2452861.05,2.06,2.06,Radial Velocity,,,,,11 UMi,229.275,71.8238889,5.02,,,,,119.5,0.04,1.8,24.08,K4III,1.56,4340.0,,
```

Response:
An easy way to do this is using the pandas library like this.
```
import pandas as pd
fields = ['star_name', 'ra']

df = pd.read_csv('data.csv', skipinitialspace=True, usecols=fields)
# See the keys
print df.keys()
# See content in 'star_name'
print df.star_name
```
The problem here was the skipinitialspace which remove the spaces in the header. So ' star_name' becomes 'star_name'


Question:
Pandas Groupby Range of Values Is there an easy method in pandas to invoke groupby on a range of values increments? For instance given the example below can I bin and group column B with a 0.155 increment so that for example, the first couple of groups in column B are divided into ranges between '0 - 0.155, 0.155 - 0.31 ...`
```
import numpy as np
import pandas as pd
df=pd.DataFrame({'A':np.random.random(20),'B':np.random.random(20)})

     A         B
0  0.383493  0.250785
1  0.572949  0.139555
2  0.652391  0.401983
3  0.214145  0.696935
4  0.848551  0.516692
```
Alternatively I could first categorize the data by those increments into a new column and subsequently use groupby to determine any relevant statistics that may be applicable in column A?

Response:
You might be interested in pd.cut:
```

```python
df.groupby(pd.cut(df["B"], np.arange(0, 1.0+0.155, 0.155))).sum()
#Output
#                      A         B
#B
#(0, 0.155]     2.775458  0.246394
#(0.155, 0.31]  1.123989  0.471618
#(0.31, 0.465]  2.051814  1.882763
#(0.465, 0.62]  2.277960  1.528492
#(0.62, 0.775]  1.577419  2.810723
#(0.775, 0.93]  0.535100  1.694955
#(0.93, 1.085]       NaN       NaN
```

[7 rows x 2 columns]
```


Question:
Find percentile stats of a given column I have a pandas data frame my_df, where I can find the mean(), median(), mode() of a given column:
```
my_df['field_A'].mean()
my_df['field_A'].median()
my_df['field_A'].mode()
```
I am wondering is it possible to find more detailed statistics such as the 90th percentile?

Response:
You can use the pandas.DataFrame.quantile() function. If you look at the API for quantile(), you will see it takes an argument for how to do interpolation. If you want a quantile that falls between two positions in your data: 'linear', 'lower', 'higher', 'midpoint', or 'nearest'. By default, it performs linear interpolation. These interpolation methods are discussed in the Wikipedia article for percentile
```py
import pandas as pd
import numpy as np

# sample data
np.random.seed(2023)  # for reproducibility
data = {'Category': np.random.choice(['hot', 'cold'], size=(10,)),
        'field_A': np.random.randint(0, 100, size=(10,)),
        'field_B': np.random.randint(0, 100, size=(10,))}
df = pd.DataFrame(data)

df.field_A.mean()  # Same as df['field_A'].mean()
# 51.1

df.field_A.median()
# 50.0

# You can call `quantile(i)` to get the i'th quantile,
# where `i` should be a fractional number.

df.field_A.quantile(0.1)  # 10th percentile
# 15.6

df.field_A.quantile(0.5)  # same as median
# 50.0

df.field_A.quantile(0.9)  # 90th percentile
# 88.8

df.groupby('Category').field_A.quantile(0.1)
#Category
#cold    28.8
#hot      8.6
#Name: field_A, dtype: float64
```
df
```none
Category  field_A  field_B
0     cold       96       58
1     cold       22       28
2      hot       17       81
3     cold       53       71
4     cold       47       63
5      hot       77       48
6     cold       39       32
7      hot       69       29
8      hot       88       49
9      hot        3       49
```


Question:
Remove Unnamed columns in pandas dataframe [duplicate] This question already has answers here: How to get rid of "Unnamed: 0" column in a pandas DataFrame read in from CSV file? (13 answers) Closed 6 years ago. I have a data file from columns A-G like below but when I am reading it with pd.read_csv('data.csv') it prints an extra unnamed column at the end for no reason.
```
colA    ColB    colC    colD    colE    colF    colG    Unnamed: 7
44      45      26      26      40      26      46        NaN
47      16      38      47      48      22      37        NaN
19      28      36      18      40      18      46        NaN
50      14      12      33      12      44      23        NaN
39      47      16      42      33      48      38        NaN
```
I have seen my data file various times but I have no extra data in any other column. How I should remove this extra column while reading ? Thanks

Response:
```
df = df.loc[:, ~df.columns.str.contains('^Unnamed')]

```python
df
#Output
#colA  ColB  colC  colD  colE  colF  colG
#0    44    45    26    26    40    26    46
#1    47    16    38    47    48    22    37
#2    19    28    36    18    40    18    46
#3    50    14    12    33    12    44    23
#4    39    47    16    42    33    48    38
#```
#NOTE: very often there is only one unnamed column Unnamed: 0, which is the first column in the CSV file. This is the result of the following steps: a DataFrame is saved into a CSV file using parameter index=True, which is the default behaviour we read this CSV file into a DataFrame using pd.read_csv() without explicitly specifying index_col=0 (default: index_col=None) The easiest way to get rid of this column is to specify the parameter pd.read_csv(..., index_col=0):
#```
#df = pd.read_csv('data.csv', index_col=0)
#```
```


Question:
Shift column in pandas dataframe up by one? I've got a pandas dataframe. I want to 'lag' one of my columns. Meaning, for example, shifting the entire column 'gdp' up by one, and then removing all the excess data at the bottom of the remaining rows so that all columns are of equal length again.
```none
df =
    y  gdp  cap
0   1    2    5
1   2    3    9
2   8    7    2
3   3    4    7
4   6    7    7

df_lag =
    y  gdp  cap
0   1    3    5
1   2    7    9
2   8    4    2
3   3    7    7
```
Anyway to do this?

Response:
```

```python
df['gdp'] = df['gdp'].shift(-1)
```

```python
df
#Output
#y  gdp  cap
#0  1    3    5
#1  2    7    9
#2  8    4    2
#3  3    7    7
#4  6  NaN    7
```

```python
df[:-1]
#Output
#y  gdp  cap
#0  1    3    5
#1  2    7    9
#2  8    4    2
#3  3    7    7
#```
```


Question:
What is the difference between pandas.qcut and pandas.cut? The documentation says: http://pandas.pydata.org/pandas-docs/dev/basics.html "Continuous values can be discretized using the cut (bins based on values) and qcut (bins based on sample quantiles) functions" Sounds very abstract to me... I can see the differences in the example below but what does qcut (sample quantile) actually do/mean? When would you use qcut versus cut? Thanks.
```
factors = np.random.randn(30)

```python
pd.cut(factors, 5)
#Output
#[(-0.411, 0.575], (-0.411, 0.575], (-0.411, 0.575], (-0.411, 0.575], (0.575, 1.561], ..., (-0.411, 0.575], (-1.397, -0.411], (0.575, 1.561], (-2.388, -1.397], (-0.411, 0.575]]
#Length: 30
#Categories (5, object): [(-2.388, -1.397] < (-1.397, -0.411] < (-0.411, 0.575] < (0.575, 1.561] < (1.561, 2.547]]
```

```python
pd.qcut(factors, 5)
#Output
#[(-0.348, 0.0899], (-0.348, 0.0899], (0.0899, 1.19], (0.0899, 1.19], (0.0899, 1.19], ..., (0.0899, 1.19], (-1.137, -0.348], (1.19, 2.547], [-2.383, -1.137], (-0.348, 0.0899]]
#Length: 30
#Categories (5, object): [[-2.383, -1.137] < (-1.137, -0.348] < (-0.348, 0.0899] < (0.0899, 1.19] < (1.19, 2.547]]`
#```
```

Response:
To begin, note that quantiles is just the most general term for things like percentiles, quartiles, and medians. You specified five bins in your example, so you are asking qcut for quintiles. So, when you ask for quintiles with qcut, the bins will be chosen so that you have the same number of records in each bin. You have 30 records, so should have 6 in each bin (your output should look like this, although the breakpoints will differ due to the random draw):
```
pd.qcut(factors, 5).value_counts()

[-2.578, -0.829]    6
(-0.829, -0.36]     6
(-0.36, 0.366]      6
(0.366, 0.868]      6
(0.868, 2.617]      6
```
Conversely, for cut you will see something more uneven:
```
pd.cut(factors, 5).value_counts()

(-2.583, -1.539]    5
(-1.539, -0.5]      5
(-0.5, 0.539]       9
(0.539, 1.578]      9
(1.578, 2.617]      2
```
That's because cut will choose the bins to be evenly spaced according to the values themselves and not the frequency of those values. Hence, because you drew from a random normal, you'll see higher frequencies in the inner bins and fewer in the outer. This is essentially going to be a tabular form of a histogram (which you would expect to be fairly bell shaped with 30 records).


Question: Can I assign a reset index a name? Normally when a dataframe undergoes a reset_index() the new column is assigned the name index or level_i depending on the level. Is it possible to assign the new column a name?

Response:
You can call rename on the returned df from reset_index:
```

```python
# create a df
df = pd.DataFrame(np.random.randn(5,3))
df
#Output
#0         1         2
#0 -2.845811 -0.182439 -0.526785
#1 -0.112547  0.661461  0.558452
#2  0.587060 -1.232262 -0.997973
#3 -1.009378 -0.062442  0.125875
#4 -1.129376  3.282447 -0.403731
#```
#Set the index name
#```
```

```python
df.index = df.index.set_names(['foo'])
df
#Output
#0         1         2
#foo
#0   -2.845811 -0.182439 -0.526785
#1   -0.112547  0.661461  0.558452
#2    0.587060 -1.232262 -0.997973
#3   -1.009378 -0.062442  0.125875
#4   -1.129376  3.282447 -0.403731
#```
#call reset_index and chain with rename:
#```
```

```python
df.reset_index().rename(columns={df.index.name:'bar'})
#Output
#bar         0         1         2
#0    0 -2.845811 -0.182439 -0.526785
#1    1 -0.112547  0.661461  0.558452
#2    2  0.587060 -1.232262 -0.997973
#3    3 -1.009378 -0.062442  0.125875
#4    4 -1.129376  3.282447 -0.403731
#```
#Thanks to @ayhan alternatively you can use rename_axis to rename the index prior to reset_index:
#```
```

```python
df.rename_axis('bar').reset_index()
#Output
#bar         0         1         2
#0    0 -2.845811 -0.182439 -0.526785
#1    1 -0.112547  0.661461  0.558452
#2    2  0.587060 -1.232262 -0.997973
#3    3 -1.009378 -0.062442  0.125875
#4    4 -1.129376  3.282447 -0.403731
#```
#or just overwrite the index name directly first:
#```
#df.index.name = 'bar'
#```
#and then call reset_index
```


Question:
Way to read first few lines for pandas dataframe Is there a built-in way to use read_csv to read only the first n lines of a file without knowing the length of the lines ahead of time? I have a large file that takes a long time to read, and occasionally only want to use the first, say, 20 lines to get a sample of it (and prefer not to load the full thing and take the head of it). If I knew the total number of lines I could do something like footer_lines = total_lines - n and pass this to the skipfooter keyword arg. My current solution is to manually grab the first n lines with python and StringIO it to pandas:
```
import pandas as pd
from StringIO import StringIO

n = 20
with open('big_file.csv', 'r') as f:
    head = ''.join(f.readlines(n))

df = pd.read_csv(StringIO(head))
```
It's not that bad, but is there a more concise, 'pandasic' (?) way to do it with keywords or something?

Response:
I think you can use the nrows parameter. From the docs:
```
nrows : int, default None

    Number of rows of file to read. Useful for reading pieces of large files
```
which seems to work. Using one of the standard large test files (988504479 bytes, 5344499 lines):
```

```python
import pandas as pd
```

```python
time z = pd.read_csv("P00000001-ALL.csv", nrows=20)
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 0.00 s
```

```python
len(z)
#Output
#20
```

```python
time z = pd.read_csv("P00000001-ALL.csv")
CPU times: user 27.63 s, sys: 1.92 s, total: 29.55 s
Wall time: 30.23 s
```
```


Question:
Add Leading Zeros to Strings in Pandas Dataframe I have a pandas data frame where the first 3 columns are strings:
```
ID        text1    text 2
0       2345656     blah      blah
1          3456     blah      blah
2        541304     blah      blah
3        201306       hi      blah
4   12313201308    hello      blah
```
I want to add leading zeros to the ID:
```
ID    text1    text 2
0  000000002345656     blah      blah
1  000000000003456     blah      blah
2  000000000541304     blah      blah
3  000000000201306       hi      blah
4  000012313201308    hello      blah
```
I have tried:
```
df['ID'] = df.ID.zfill(15)
df['ID'] = '{0:0>15}'.format(df['ID'])
```

Response:
str attribute contains most of the methods in string.
```
df['ID'] = df['ID'].str.zfill(15)
```
See more: http://pandas.pydata.org/pandas-docs/stable/text.html


Question:
start index at 1 for Pandas DataFrame [duplicate] This question already has answers here: Start row index from 1 instead of zero without creating additional column in pandas (6 answers) Closed 1 year ago. I need the index to start at 1 rather than 0 when writing a Pandas DataFrame to CSV. Here's an example:
```

```python
import pandas as pd
```

```python
result = pd.DataFrame({'Count': [83, 19, 20]})
```

```python
result.to_csv('result.csv', index_label='Event_id')
```
Which produces the following output:
```
```

```python
!cat result.csv
Event_id,Count
0,83
1,19
2,20
```
But my desired output is this:
```
```

```python
!cat result2.csv
Event_id,Count
1,83
2,19
3,20
```
I realize that this could be done by adding a sequence of integers shifted by 1 as a column to my data frame, but I'm new to Pandas and I'm wondering if a cleaner way exists.
```

Response:
Index is an object, and default index starts from 0:
```

```python
result.index
#Output
#Int64Index([0, 1, 2], dtype=int64)
#```
#You can shift this index by 1 with
#```
```

```python
result.index += 1
```

```python
result.index
#Output
#Int64Index([1, 2, 3], dtype=int64)
#```
```


Question:
group by in group by and average I have a dataframe like this:
```
cluster  org      time
   1      a       8
   1      a       6
   2      h       34
   1      c       23
   2      d       74
   3      w       6
```
I would like to calculate the average of time per org per cluster. Expected result:
```
cluster mean(time)
1       15 #=((8 + 6) / 2 + 23) / 2
2       54 #=(74 + 34) / 2
3       6
```
I do not know how to do it in Pandas, can anybody help?

Response:
If you want to first take mean on the combination of ['cluster', 'org'] and then take mean on cluster groups, you can use:
```

```python
(df.groupby(['cluster', 'org'], as_index=False).mean()
            .groupby('cluster')['time'].mean())
#Output
#cluster
#1          15
#2          54
#3           6
#Name: time, dtype: int64
#```
#If you want the mean of cluster groups only, then you can use:
#```
```

```python
df.groupby(['cluster']).mean()
#Output
#time
#cluster
#1        12.333333
#2        54.000000
#3         6.000000
#```
#You can also use groupby on ['cluster', 'org'] and then use mean():
#```
```

```python
df.groupby(['cluster', 'org']).mean()
#Output
#time
#cluster org
#1       a    438886
#        c        23
#2       d      9874
#        h        34
#3       w         6
#```
```


Question:
Pandas: convert categories to numbers Suppose I have a dataframe with countries that goes as:
```
cc | temp
US | 37.0
CA | 12.0
US | 35.0
AU | 20.0
```
I know that there is a pd.get_dummies function to convert the countries to 'one-hot encodings'. However, I wish to convert them to indices instead such that I will get cc_index = [1,2,1,3] instead. I'm assuming that there is a faster way than using the get_dummies along with a numpy where clause as shown below: [np.where(x) for x in df.cc.get_dummies().values] This is somewhat easier to do in R using 'factors' so I'm hoping pandas has something similar.

Response:
First, change the type of the column:
```
df.cc = pd.Categorical(df.cc)
```
Now the data look similar but are stored categorically. To capture the category codes:
```
df['code'] = df.cc.codes
```
Now you have:
```
cc  temp  code
0  US  37.0     2
1  CA  12.0     1
2  US  35.0     2
3  AU  20.0     0
```
If you don't want to modify your DataFrame but simply get the codes:
```
df.cc.astype('category').codes
```
Or use the categorical column as an index:
```
df2 = pd.DataFrame(df.temp)
df2.index = pd.CategoricalIndex(df.cc)
```


Question:
Group dataframe and get sum and count I have a dataframe that looks like this:
```
Company Name              Organisation Name  Amount
10118  Vifor Pharma UK Ltd  Welsh Assoc for Gastro & Endo 2700.00
10119  Vifor Pharma UK Ltd    Welsh IBD Specialist Group,  169.00
10120  Vifor Pharma UK Ltd             West Midlands AHSN 1200.00
10121  Vifor Pharma UK Ltd           Whittington Hospital   63.00
10122  Vifor Pharma UK Ltd                 Ysbyty Gwynedd   75.93
```
How do I sum the Amount and count the Organisation Name, to get a new dataframe that looks like this?
```
Company Name             Organisation Count   Amount
10118  Vifor Pharma UK Ltd                              5 11000.00
```
I know how to sum or count:
```
df.groupby('Company Name').sum()
df.groupby('Company Name').count()
```
But not how to do both!

Response:
try this:
```

```python
(df.groupby('Company Name')
   .....:    .agg({'Organisation Name':'count', 'Amount': 'sum'})
   .....:    .reset_index()
   .....:    .rename(columns={'Organisation Name':'Organisation Count'})
   .....: )
#Output
#Company Name   Amount  Organisation Count
#0  Vifor Pharma UK Ltd  4207.93                   5
#```
#or if you don't want to reset index:
#```
#df.groupby('Company Name')['Amount'].agg(['sum','count'])
#```
#or
#```
#df.groupby('Company Name').agg({'Amount': ['sum','count']})
#```
#Demo:
#```
```

```python
df.groupby('Company Name')['Amount'].agg(['sum','count'])
#Output
#sum  count
#Company Name
#Vifor Pharma UK Ltd  4207.93      5
```

```python
df.groupby('Company Name').agg({'Amount': ['sum','count']})
#Output
#Amount
#                         sum count
#Company Name
#Vifor Pharma UK Ltd  4207.93     5
#```
```


Question:
pandas dataframe select columns in multiindex [duplicate] This question already has answers here: Selecting columns from pandas MultiIndex (13 answers) Closed 6 years ago. I have the following pd.DataFrame:
```
Name    0                       1                      ...
Col     A           B           A            B         ...
0       0.409511    -0.537108   -0.355529    0.212134  ...
1       -0.332276   -1.087013    0.083684    0.529002  ...
2       1.138159    -0.327212    0.570834    2.337718  ...
```
It has MultiIndex columns with names=['Name', 'Col'] and hierarchical levels. The Name label goes from 0 to n, and for each label, there are two A and B columns. I would like to subselect all the A (or B) columns of this DataFrame.

Response:
There is a get_level_values method that you can use in conjunction with boolean indexing to get the the intended result.
```

```python
df = pd.DataFrame(np.random.random((4,4)))
df.columns = pd.MultiIndex.from_product([[1,2],['A','B']])
print df
          1                   2
          A         B         A         B
0  0.543980  0.628078  0.756941  0.698824
1  0.633005  0.089604  0.198510  0.783556
2  0.662391  0.541182  0.544060  0.059381
3  0.841242  0.634603  0.815334  0.848120
```

```python
print df.iloc[:, df.columns.get_level_values(1)=='A']
          1         2
          A         A
0  0.543980  0.756941
1  0.633005  0.198510
2  0.662391  0.544060
3  0.841242  0.815334
```
```


Question:
Convert dataframe index to datetime How do I convert a pandas index of strings to datetime format? My dataframe df is like this:
```none
value
2015-09-25 00:46    71.925000
2015-09-25 00:47    71.625000
2015-09-25 00:48    71.333333
2015-09-25 00:49    64.571429
2015-09-25 00:50    72.285714
```
but the index is of type string, but I need it a datetime format because I get the error:
```none
'Index' object has no attribute 'hour'
```
when using
```py
df["A"] = df.index.hour
```

Response:
It should work as expected. Try to run the following example.
```
import pandas as pd
import io

data = """value
"2015-09-25 00:46"    71.925000
"2015-09-25 00:47"    71.625000
"2015-09-25 00:48"    71.333333
"2015-09-25 00:49"    64.571429
"2015-09-25 00:50"    72.285714"""

df = pd.read_table(io.StringIO(data), delim_whitespace=True)

# Converting the index as date
df.index = pd.to_datetime(df.index)

# Extracting hour & minute
df['A'] = df.index.hour
df['B'] = df.index.minute
df

#                          value  A   B
# 2015-09-25 00:46:00  71.925000  0  46
# 2015-09-25 00:47:00  71.625000  0  47
# 2015-09-25 00:48:00  71.333333  0  48
# 2015-09-25 00:49:00  64.571429  0  49
# 2015-09-25 00:50:00  72.285714  0  50
```


Question:
Numpy isnan() fails on an array of floats (from pandas dataframe apply) I have an array of floats (some normal numbers, some nans) that is coming out of an apply on a pandas dataframe. For some reason, numpy.isnan is failing on this array, however as shown below, each element is a float, numpy.isnan runs correctly on each element, the type of the variable is definitely a numpy array. What's going on?!
```
set([type(x) for x in tester])
Out[59]: {float}

tester
Out[60]:
array([-0.7000000000000001, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
   nan, nan], dtype=object)

set([type(x) for x in tester])
Out[61]: {float}

np.isnan(tester)
Traceback (most recent call last):

File "<ipython-input-62-e3638605b43c>", line 1, in <module>
np.isnan(tester)

TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

set([np.isnan(x) for x in tester])
Out[65]: {False, True}

type(tester)
Out[66]: numpy.ndarray
```

Response:
np.isnan can be applied to NumPy arrays of native dtype (such as np.float64):
```

```python
np.isnan(np.array([np.nan, 0], dtype=np.float64))
#Output
#array([ True, False], dtype=bool)
#```
#but raises TypeError when applied to object arrays:
#```
```

```python
np.isnan(np.array([np.nan, 0], dtype=object))
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
```
Since you have Pandas, you could use pd.isnull instead -- it can accept NumPy arrays of object or native dtypes:
```
```

```python
pd.isnull(np.array([np.nan, 0], dtype=float))
#Output
#array([ True, False], dtype=bool)
```

```python
pd.isnull(np.array([np.nan, 0], dtype=object))
#Output
#array([ True, False], dtype=bool)
#```
#Note that None is also considered a null value in object arrays.
```


Question:
Converting between datetime and Pandas Timestamp objects I have the following:
```
> date1
Timestamp('2014-01-23 00:00:00', tz=None)

> date2
datetime.date(2014, 3, 26)
```
and I read on this answer that I could use pandas.to_datetime() to convert from Timestamps to datetime objects, but it doesn't seem to work:
```
> pd.to_datetime(date1)
Timestamp('2014-01-23 00:00:00', tz=None)
```
Why? How can I convert between these two formats?

Response:
You can use the to_pydatetime method to be more explicit:
```

```python
ts = pd.Timestamp('2014-01-23 00:00:00', tz=None)
```

```python
ts.to_pydatetime()
#Output
#datetime.datetime(2014, 1, 23, 0, 0)
#```
#It's also available on a DatetimeIndex:
#```
```

```python
rng = pd.date_range('1/10/2011', periods=3, freq='D')
```

```python
rng.to_pydatetime()
#Output
#array([datetime.datetime(2011, 1, 10, 0, 0),
#       datetime.datetime(2011, 1, 11, 0, 0),
#       datetime.datetime(2011, 1, 12, 0, 0)], dtype=object)
#```
```


Question:
re.sub erroring with "Expected string or bytes-like object" I have read multiple posts regarding this error, but I still can't figure it out. When I try to loop through my function:
```py
def fix_Plan(location):
    letters_only = re.sub("[^a-zA-Z]",  # Search for all non-letters
                          " ",          # Replace all non-letters with spaces
                          location)     # Column and row to search

    words = letters_only.lower().split()
    stops = set(stopwords.words("english"))
    meaningful_words = [w for w in words if not w in stops]
    return (" ".join(meaningful_words))

col_Plan = fix_Plan(train["Plan"][0])
num_responses = train["Plan"].size
clean_Plan_responses = []

for i in range(0,num_responses):
    clean_Plan_responses.append(fix_Plan(train["Plan"][i]))
```
Here is the error:
```none
Traceback (most recent call last):
  File "C:/Users/xxxxx/PycharmProjects/tronc/tronc2.py", line 48, in <module>
    clean_Plan_responses.append(fix_Plan(train["Plan"][i]))
  File "C:/Users/xxxxx/PycharmProjects/tronc/tronc2.py", line 22, in fix_Plan
    location)  # Column and row to search
  File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python36\lib\re.py", line 191, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object
```

Response:
As you stated in the comments, some of the values appeared to be floats, not strings. You will need to change it to strings before passing it to re.sub. The simplest way is to change location to str(location) when using re.sub. It wouldn't hurt to do it anyways even if it's already a str.
```
letters_only = re.sub("[^a-zA-Z]",  # Search for all non-letters
                          " ",          # Replace all non-letters with spaces
                          str(location))
```


Question:
Getting vertical gridlines to appear in line plot in matplotlib I want to get both horizontal and vertical grid lines on my plot but only the horizontal grid lines are appearing by default. I am using a pandas.DataFrame from an sql query in python to generate a line plot with dates on the x-axis. I'm not sure why they do not appear on the dates and I have tried to search for an answer to this but couldn't find one. All I have used to plot the graph is the simple code below.
```
data.plot()
grid('on')
```
data is the DataFrame which contains the dates and the data from the sql query. I have also tried adding the code below but I still get the same output with no vertical grid lines.
```
ax = plt.axes()
ax.yaxis.grid() # horizontal lines
ax.xaxis.grid() # vertical lines
```
Any suggestions?

Response:
You may need to give boolean arg in your calls, e.g. use ax.yaxis.grid(True) instead of ax.yaxis.grid(). Additionally, since you are using both of them you can combine into ax.grid, which works on both, rather than doing it once for each dimension.
```
ax = plt.gca()
ax.grid(True)
```
That should sort you out.