Question:
removing time from date&time variable in pandas? i have a variable consisting of 300k records with dates and the date look like 2015-02-21 12:08:51 from that date i want to remove time type of date variable is pandas.core.series.series This is the way i tried
```
from datetime import datetime,date
date_str = textdata['vfreceiveddate']
format_string = "%Y-%m-%d"
then = datetime.strftime(date_str,format_string)
```
some Random ERROR In the above code textdata is my datasetname and vfreceived date is a variable consisting of dates How can i write the code to remove the time from the datetime.

Response:
Assuming all your datetime strings are in a similar format then just convert them to datetime using to_datetime and then call the dt.date attribute to get just the date portion:
```

```python
df = pd.DataFrame({'date':['2015-02-21 12:08:51']})
df
#Output
#date
#0  2015-02-21 12:08:51
```

```python
df['date'] = pd.to_datetime(df['date']).dt.date
df
#Output
#date
#0  2015-02-21
#```
#EDIT If you just want to change the display and not the dtype then you can call dt.normalize:
#```
```

```python
df['date'] = pd.to_datetime(df['date']).dt.normalize()
df
#Output
#date
#0 2015-02-21
#```
#You can see that the dtype remains as datetime:
#```
```

```python
df.dtypes
#Output
#date    datetime64[ns]
#dtype: object
#```
```


Question:
How to pass another entire column as argument to pandas fillna() I would like to fill missing values in one column with values from another column, using fillna method. (I read that looping through each row would be very bad practice and that it would be better to do everything in one go but I could not find out how to do it with fillna.) Data before:
```
Day  Cat1  Cat2
1    cat   mouse
2    dog   elephant
3    cat   giraf
4    NaN   ant
```
Data after:
```
Day  Cat1  Cat2
1    cat   mouse
2    dog   elephant
3    cat   giraf
4    ant   ant
```

Response:
You can provide this column to fillna (see docs), it will use those values on matching indexes to fill:
```

```python
df['Cat1'].fillna(df['Cat2'])
#Output
#0    cat
#1    dog
#2    cat
#3    ant
#Name: Cat1, dtype: object
#```
```


Question:
pandas converting floats to strings without decimals I have a dataframe
```
df = pd.DataFrame([
        ['2', '3', 'nan'],
        ['0', '1', '4'],
        ['5', 'nan', '7']
    ])

print df

   0    1    2
0  2    3  nan
1  0    1    4
2  5  nan    7
```
I want to convert these strings to numbers and sum the columns and convert back to strings. Using astype(float) seems to get me to the number part. Then summing is easy with sum(). Then back to strings should be easy too with astype(str)
```
df.astype(float).sum().astype(str)

0     7.0
1     4.0
2    11.0
dtype: object
```
That's almost what I wanted. I wanted the string version of integers. But floats have decimals. How do I get rid of them? I want this
```
0     7
1     4
2    11
dtype: object
```

Response:
For pandas >= 1.0:  type was introduced for 'Int64'. You can now do this:
```
df['your_column'].astype('Int64').astype('str')
```
And it will properly convert 1.0 to 1. Alternative: If you do not want to change the display options of all pandas, @maxymoo solution does, you can use apply:
```
df['your_column'].apply(lambda x: f'{x:.0f}')
```


Question:
Scrape tables into dataframe with BeautifulSoup I'm trying to scrape the data from the coins catalog. There is one of the pages. I need to scrape this data into Dataframe So far I have this code:
```
import bs4 as bs
import urllib.request
import pandas as pd

source = urllib.request.urlopen('http://www.gcoins.net/en/catalog/view/45518').read()
soup = bs.BeautifulSoup(source,'lxml')

table = soup.find('table', attrs={'class':'subs noBorders evenRows'})
table_rows = table.find_all('tr')

for tr in table_rows:
    td = tr.find_all('td')
    row = [tr.text for tr in td]
    print(row)                    # I need to save this data instead of printing it
```
It produces following output:
```
[]
['', '', '1882', '', '108,000', 'UNC', '—']
[' ', '', '1883', '', '786,000', 'UNC', '~ $3.99']
[' ', " \n\n\n\n\t\t\t\t\t\t\t$('subGraph55337').on('click', function(event) {\n\t\t\t\t\t\t\t\tLightview.show({\n\t\t\t\t\t\t\t\t\thref : '/en/catalog/ajax/subgraph?id=55337',\n\t\t\t\t\t\t\t\t\trel : 'ajax',\n\t\t\t\t\t\t\t\t\toptions : {\n\t\t\t\t\t\t\t\t\t\tautosize : true,\n\t\t\t\t\t\t\t\t\t\ttopclose : true,\n\t\t\t\t\t\t\t\t\t\tajax : {\n\t\t\t\t\t\t\t\t\t\t\tevalScripts : true\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t} \n\t\t\t\t\t\t\t\t});\n\t\t\t\t\t\t\t\tevent.stop();\n\t\t\t\t\t\t\t\treturn false;\n\t\t\t\t\t\t\t});\n\t\t\t\t\t\t", '1884', '', '4,604,000', 'UNC', '~ $2.08–$4.47']
[' ', '', '1885', '', '1,314,000', 'UNC', '~ $3.20']
['', '', '1886', '', '444,000', 'UNC', '—']
[' ', '', '1888', '', '413,000', 'UNC', '~ $2.88']
[' ', '', '1889', '', '568,000', 'UNC', '~ $2.56']
[' ', " \n\n\n\n\t\t\t\t\t\t\t$('subGraph55342').on('click', function(event) {\n\t\t\t\t\t\t\t\tLightview.show({\n\t\t\t\t\t\t\t\t\thref : '/en/catalog/ajax/subgraph?id=55342',\n\t\t\t\t\t\t\t\t\trel : 'ajax',\n\t\t\t\t\t\t\t\t\toptions : {\n\t\t\t\t\t\t\t\t\t\tautosize : true,\n\t\t\t\t\t\t\t\t\t\ttopclose : true,\n\t\t\t\t\t\t\t\t\t\tajax : {\n\t\t\t\t\t\t\t\t\t\t\tevalScripts : true\n\t\t\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t\t\t} \n\t\t\t\t\t\t\t\t});\n\t\t\t\t\t\t\t\tevent.stop();\n\t\t\t\t\t\t\t\treturn false;\n\t\t\t\t\t\t\t});\n\t\t\t\t\t\t", '1890', '', '2,137,000', 'UNC', '~ $1.28–$4.79']
['', '', '1891', '', '605,000', 'UNC', '—']
[' ', '', '1892', '', '205,000', 'UNC', '~ $4.47']
[' ', '', '1893', '', '754,000', 'UNC', '~ $4.79']
[' ', '', '1894', '', '532,000', 'UNC', '~ $3.20']
[' ', '', '1895', '', '423,000', 'UNC', '~ $2.40']
['', '', '1896', '', '174,000', 'UNC', '—']
```
But when I'm trying to save it to Dataframe and export to excel it contains just the last value:
```
0
0
1
2     1896
3
4  174,000
5      UNC
6        —
```

Response:
Pandas already has a built-in method to convert the table on the web to a dataframe:
```
table = soup.find_all('table')
df = pd.read_html(str(table))[0]
```


Question:
Pandas Groupby: Count and mean combined Working with pandas to try and summarise a data frame as a count of certain categories, as well as the means sentiment score for these categories. There is a table full of strings that have different sentiment scores, and I want to group each text source by saying how many posts they have, as well as the average sentiment of these posts. My (simplified) data frame looks like this:
```
source    text              sent
--------------------------------
bar       some string       0.13
foo       alt string        -0.8
bar       another str       0.7
foo       some text         -0.2
foo       more text         -0.5
```
The output from this should be something like this:
```
source    count     mean_sent
-----------------------------
foo       3         -0.5
bar       2         0.415
```
The answer is somewhere along the lines of:
```
df['sent'].groupby(df['source']).mean()
```
Yet only gives each source and it's mean, with no column headers.

Response:
You can use groupby with aggregate:
```
df = df.groupby('source') \
       .agg({'text':'size', 'sent':'mean'}) \
       .rename(columns={'text':'count','sent':'mean_sent'}) \
       .reset_index()
print (df)
  source  count  mean_sent
0    bar      2      0.415
1    foo      3     -0.500
```


Question:
Trying to merge 2 dataframes but get ValueError These are my two dataframes saved in two variables:
```
> print(df.head())
>
          club_name  tr_jan  tr_dec  year
    0  ADO Den Haag    1368    1422  2010
    1  ADO Den Haag    1455    1477  2011
    2  ADO Den Haag    1461    1443  2012
    3  ADO Den Haag    1437    1383  2013
    4  ADO Den Haag    1386    1422  2014
> print(rankingdf.head())
>
           club_name  ranking  year
    0    ADO Den Haag    12    2010
    1    ADO Den Haag    13    2011
    2    ADO Den Haag    11    2012
    3    ADO Den Haag    14    2013
    4    ADO Den Haag    17    2014
```
I'm trying to merge these two using this code:
```
new_df = df.merge(ranking_df, on=['club_name', 'year'], how='left')
```
The how='left' is added because I have less datapoints in my ranking_df than in my standard df. The expected behaviour is as such:
```
> print(new_df.head())
>

      club_name  tr_jan  tr_dec  year    ranking
0  ADO Den Haag    1368    1422  2010    12
1  ADO Den Haag    1455    1477  2011    13
2  ADO Den Haag    1461    1443  2012    11
3  ADO Den Haag    1437    1383  2013    14
4  ADO Den Haag    1386    1422  2014    17
```
But I get this error: ValueError: You are trying to merge on object and int64 columns. If you wish to proceed you should use pd.concat But I do not wish to use concat since I want to merge the trees not just add them on. Another behaviour that's weird in my mind is that my code works if I save the first df to .csv and then load that .csv into a dataframe. The code for that:
```
df = pd.DataFrame(data_points, columns=['club_name', 'tr_jan', 'tr_dec', 'year'])
df.to_csv('preliminary.csv')

df = pd.read_csv('preliminary.csv', index_col=0)

ranking_df = pd.DataFrame(rankings, columns=['club_name', 'ranking', 'year'])

new_df = df.merge(ranking_df, on=['club_name', 'year'], how='left')
```
I think that it has to do with the index_col=0 parameter. But I have no idea to fix it without having to save it, it doesn't matter much but is kind of an annoyance that I have to do that.

Response: In one of your dataframes the year is a string and the other it is an int64 you can convert it first and then join (e.g. df['year']=df['year'].astype(int) or as RafaelC suggested df.year.astype(int)) Edit: Also note the comment by Anderson Zhu: Just in case you have None or missing values in one of your dataframes, you need to use Int64 instead of int. See the reference here.


Question:
Remove unwanted parts from strings in a column I am looking for an efficient way to remove unwanted parts from strings in a DataFrame column. Data looks like:
```
time    result
1    09:00   +52A
2    10:00   +62B
3    11:00   +44a
4    12:00   +30b
5    13:00   -110a
```
I need to trim these data to:
```
time    result
1    09:00   52
2    10:00   62
3    11:00   44
4    12:00   30
5    13:00   110
```
I tried .str.lstrip('+-') and .str.rstrip('aAbBcC'), but got an error:
```
TypeError: wrapper() takes exactly 1 argument (2 given)
```
Any pointers would be greatly appreciated!

Response:
How do I remove unwanted parts from strings in a column? 6 years after the original question was posted, pandas now has a good number of "vectorised" string functions that can succinctly perform these string manipulation operations. This answer will explore some of these string functions, suggest faster alternatives, and go into a timings comparison at the end. .str.replace Specify the substring/pattern to match, and the substring to replace it with.
```
pd.__version__
# '0.24.1'

df
    time result
1  09:00   +52A
2  10:00   +62B
3  11:00   +44a
4  12:00   +30b
5  13:00  -110a
```
```
df['result'] = df['result'].str.replace(r'\D', '', regex=True)
df

    time result
1  09:00     52
2  10:00     62
3  11:00     44
4  12:00     30
5  13:00    110
```
If you need the result converted to an integer, you can use Series.astype,
```
df['result'] = df['result'].str.replace(r'\D', '', regex=True).astype(int)

df.dtypes
time      object
result     int64
dtype: object
```
If you don't want to modify df in-place, use DataFrame.assign:
```
df2 = df.assign(result=df['result'].str.replace(r'\D', '', regex=True))
df
# Unchanged
```
.str.extract Useful for extracting the substring(s) you want to keep.
```
df['result'] = df['result'].str.extract(r'(\d+)', expand=False)
df

    time result
1  09:00     52
2  10:00     62
3  11:00     44
4  12:00     30
5  13:00    110
```
With extract, it is necessary to specify at least one capture group. expand=False will return a Series with the captured items from the first capture group. ###.str.split and .str.get Splitting works assuming all your strings follow this consistent structure.
```
# df['result'] = df['result'].str.split(r'\D').str[1]
df['result'] = df['result'].str.split(r'\D').str.get(1)
df

    time result
1  09:00     52
2  10:00     62
3  11:00     44
4  12:00     30
5  13:00    110
```
Do not recommend if you are looking for a general solution. If you are satisfied with the succinct and readable str accessor-based solutions above, you can stop here. However, if you are interested in faster, more performant alternatives, keep reading. Optimizing: List Comprehensions In some circumstances, list comprehensions should be favoured over pandas string functions. The reason is because string functions are inherently hard to vectorize (in the true sense of the word), so most string and regex functions are only wrappers around loops with more overhead. My write-up, Are for-loops in pandas really bad? When should I care?, goes into greater detail. The str.replace option can be re-written using re.sub
```
import re

# Pre-compile your regex pattern for more performance.
p = re.compile(r'\D')
df['result'] = [p.sub('', x) for x in df['result']]
df

    time result
1  09:00     52
2  10:00     62
3  11:00     44
4  12:00     30
5  13:00    110
```
The str.extract example can be re-written using a list comprehension with re.search,
```
p = re.compile(r'\d+')
df['result'] = [p.search(x)[0] for x in df['result']]
df

    time result
1  09:00     52
2  10:00     62
3  11:00     44
4  12:00     30
5  13:00    110
```
If NaNs or no-matches are a possibility, you will need to re-write the above to include some error checking. I do this using a function.
```
def try_extract(pattern, string):
    try:
        m = pattern.search(string)
        return m.group(0)
    except (TypeError, ValueError, AttributeError):
        return np.nan

p = re.compile(r'\d+')
df['result'] = [try_extract(p, x) for x in df['result']]
df

    time result
1  09:00     52
2  10:00     62
3  11:00     44
4  12:00     30
5  13:00    110
```
We can also re-write @eumiro's and @MonkeyButter's answers using list comprehensions:
```
df['result'] = [x.lstrip('+-').rstrip('aAbBcC') for x in df['result']]
```
And,
```
df['result'] = [x[1:-1] for x in df['result']]
```
Same rules for handling NaNs, etc, apply. Performance Comparison Graphs generated using perfplot. Full code listing, for your reference. The relevant functions are listed below. Some of these comparisons are unfair because they take advantage of the structure of OP's data, but take from it what you will. One thing to note is that every list comprehension function is either faster or comparable than its equivalent pandas variant. Functions
```
def eumiro(df):
    return df.assign(
        result=df['result'].map(lambda x: x.lstrip('+-').rstrip('aAbBcC')))

def coder375(df):
    return df.assign(
        result=df['result'].replace(r'\D', r'', regex=True))

def monkeybutter(df):
    return df.assign(result=df['result'].map(lambda x: x[1:-1]))

def wes(df):
    return df.assign(result=df['result'].str.lstrip('+-').str.rstrip('aAbBcC'))

def cs1(df):
    return df.assign(result=df['result'].str.replace(r'\D', ''))

def cs2_ted(df):
    # `str.extract` based solution, similar to @Ted Petrou's. so timing together.
    return df.assign(result=df['result'].str.extract(r'(\d+)', expand=False))

def cs1_listcomp(df):
    return df.assign(result=[p1.sub('', x) for x in df['result']])

def cs2_listcomp(df):
    return df.assign(result=[p2.search(x)[0] for x in df['result']])

def cs_eumiro_listcomp(df):
    return df.assign(
        result=[x.lstrip('+-').rstrip('aAbBcC') for x in df['result']])

def cs_mb_listcomp(df):
    return df.assign(result=[x[1:-1] for x in df['result']])
```


Question:
How to implement a Boolean search with multiple columns in pandas I have a pandas df and would like to accomplish something along these lines (in SQL terms):
```
SELECT * FROM df WHERE column1 = 'a' OR column2 = 'b' OR column3 = 'c' etc.
```
Now this works, for one column/value pair:
```
foo = df.loc[df['column']==value]
```
However, I'm not sure how to expand that to multiple column/value pairs. To be clear, each column matches a different value.

Response:
You need to enclose multiple conditions in braces due to operator precedence and use the bitwise and (&) and or (|) operators:
```
foo = df[(df['column1']==value) | (df['columns2'] == 'b') | (df['column3'] == 'c')]
```
If you use and or or, then pandas is likely to moan that the comparison is ambiguous. In that case, it is unclear whether we are comparing every value in a series in the condition, and what does it mean if only 1 or all but 1 match the condition. That is why you should use the bitwise operators or the numpy np.all or np.any to specify the matching criteria. There is also the query method: http://pandas.pydata.org/pandas-docs/dev/generated/pandas.DataFrame.query.html but there are some limitations mainly to do with issues where there could be ambiguity between column names and index values.


Question:
How to set dtypes by column in pandas DataFrame I want to bring some data into a pandas DataFrame and I want to assign dtypes for each column on import. I want to be able to do this for larger datasets with many different columns, but, as an example:
```
myarray = np.random.randint(0,5,size=(2,2))
mydf = pd.DataFrame(myarray,columns=['a','b'], dtype=[float,int])
mydf.dtypes
```
results in: TypeError: data type not understood I tried a few other methods such as:
```
mydf = pd.DataFrame(myarray,columns=['a','b'], dtype={'a': int})
```
TypeError: object of type 'type' has no len() If I put dtype=(float,int) it applies a float format to both columns. In the end I would like to just be able to pass it a list of datatypes the same way I can pass it a list of column names.

Response:
I just ran into this, and the pandas issue is still open, so I'm posting my workaround. Assuming df is my DataFrame and dtype is a dict mapping column names to types:
```
for k, v in dtype.items():
    df[k] = df[k].astype(v)
```
(note: use dtype.iteritems() in python 2) For the reference: The list of allowed data types (NumPy dtypes): https://docs.scipy.org/doc/numpy-1.12.0/reference/arrays.dtypes.html Pandas also supports some other types. E.g., category: http://pandas.pydata.org/pandas-docs/stable/categorical.html The relevant GitHub issue: https://github.com/pandas-dev/pandas/issues/9287


Question:
Creating dataframe from a dictionary where entries have different lengths Say I have a dictionary with 10 key-value pairs. Each entry holds a numpy array. However, the length of the array is not the same for all of them. How can I create a dataframe where each column holds a different entry? When I try:
```py
import pandas as pd
import numpy as np
from string import ascii_uppercase  # from the standard library

# repeatable sample data
np.random.seed(2023)
data = {k: np.random.randn(v) for k, v in zip(ascii_uppercase[:10], range(10, 20))}

df = pd.DataFrame(data)
```
I get:
```py
ValueError: arrays must all be the same length
```
Any way to overcome this? I am happy to have Pandas use NaN to pad those columns for the shorter entries. Desired Result
```none
A         B         C         D         E         F         G         H         I         J
0   0.711674 -1.076522 -1.502178 -1.519748  0.340619  0.051132  0.036537  0.367296  1.056500 -1.186943
1  -0.324485 -0.325682 -1.379593  2.097329 -1.253501 -0.238061  2.431822 -0.576828 -0.733918 -0.540638
2  -1.001871 -1.035498 -0.204455  0.892562  0.370788 -0.208009  0.422599 -0.416005 -0.083968 -0.638495
3   0.236251 -0.426320  0.642125  1.596488  0.455254  0.401304  1.843922 -0.137542  0.127288  0.150411
4  -0.102160 -1.029361 -0.181176 -0.638762 -2.283720  0.183169 -0.221562  1.294987  0.344423  0.919450
5  -1.141293 -0.521774  0.771749 -1.133047 -0.000822  1.235830  0.337117  0.520589  0.685970  0.910146
6   2.654407 -0.422758  0.741523  0.656597  2.398876 -0.291800 -0.557180 -0.194273  0.399908  1.605234
7   1.440605 -0.099244  1.324763  0.595787 -2.583105  0.029992  0.053141 -0.385593  0.893458  0.667165
8   0.098902 -1.380258  0.439287 -0.811120  1.311009 -0.868404  1.053804 -3.065784  0.384793  0.950338
9  -3.121532  0.301903 -0.557873 -0.300535 -1.579478  0.604346 -0.658515 -0.668181  0.641113  0.734329
10       NaN -1.033599  0.927080  1.008391 -0.840683  0.728554  1.844449  0.056965 -0.577314  1.015465
11       NaN       NaN -0.600727 -1.087762 -0.165509  1.364820 -0.075514 -0.909368 -0.819947  0.627386
12       NaN       NaN       NaN -1.787079 -2.068410  1.342694  0.264263 -1.487910  0.746819  1.062655
13       NaN       NaN       NaN       NaN  0.452739 -1.456708 -1.395359  1.169611  1.836805  0.262885
14       NaN       NaN       NaN       NaN       NaN  0.969357  0.708416  0.393677 -1.455490 -2.086486
15       NaN       NaN       NaN       NaN       NaN       NaN  0.762756  0.530569 -0.828721 -1.076369
16       NaN       NaN       NaN       NaN       NaN       NaN       NaN -0.586429 -0.609144 -0.507519
17       NaN       NaN       NaN       NaN       NaN       NaN       NaN       NaN -1.071297 -0.274501
18       NaN       NaN       NaN       NaN       NaN       NaN       NaN       NaN       NaN  1.848811
```

Response:
In Python 3.x:
```py
import pandas as pd
import numpy as np

d = dict( A = np.array([1,2]), B = np.array([1,2,3,4]) )

pd.DataFrame(dict([ (k,pd.Series(v)) for k,v in d.items() ]))

Out[7]:
    A  B
0   1  1
1   2  2
2 NaN  3
3 NaN  4
```
In Python 2.x: replace d.items() with d.iteritems().


Question: Pandas: create dataframe from list of namedtuple I'm new to pandas, therefore perhaps I'm asking a very stupid question. Normally initialization of data frame in pandas would be column-wise, where I put in dict with key of column names and values of list-like object with same length. But I would love to initialize row-wise without dynamically concat-ing rows. Say I have a list of namedtuple, is there a optimized operation that will give me a pandas data frame directly from it?

Response:
The function you want is from_records. For namedtuple instances you must pass the _fields property of the namedtuple to the columns parameter of from_records, in addition to a list of namedtuples:
```
df = pd.DataFrame.from_records(
   [namedtuple_instance1, namedtuple_instance2],
   columns=namedtuple_type._fields
)
```
If you have dictionaries, you can use it directly as
```
df = pd.DataFrame.from_records([dict(a=1, b=2), dict(a=2, b=3)])
```


Question:
Good alternative to Pandas .append() method, now that it has been deprecated? I use the following method a lot to append a single row to a dataframe. However, it has been deprecated. One thing I really like about it is that it allows you to append a simple dict object. For example:
```
# Creating an empty dataframe
df = pd.DataFrame(columns=['a', 'b'])

# Appending a row
df = df.append({ 'a': 1, 'b': 2 }, ignore_index=True)
```
Again, what I like most about this is that the code is very clean and requires very few lines. Now I suppose the recommended alternative is:
```
# Create the new row as its own dataframe
df_new_row = pd.DataFrame({ 'a': [1], 'b': [2] })
df = pd.concat([df, df_new_row])
```
So what was one line of code before is now two lines with a throwaway variable and extra cruft where I create the new dataframe. :( Is there a good way to do this that just uses a dict like I have in the past (that is not deprecated)?

Response:
I also like the append method. But you can do it in one line with a list of dicts
```
df = pd.concat([df, pd.DataFrame.from_records([{ 'a': 1, 'b': 2 }])])
```
or using loc and tuples for values on DataFrames with incremenal ascending indexes
```
df.loc[len(df), ['a','b']] = 1, 2
```
or maybe
```
df.loc[len(df), df.columns] = 3, 4
```


Question:
Excel file format cannot be determined, you must specify an engine manually I am not sure why I am getting this error although sometimes my code works fine! Excel file format cannot be determined, you must specify an engine manually. Here below is my code with steps: 1- list of columns of customers Id:
```
customer_id = ["ID","customer_id","consumer_number","cus_id","client_ID"]
```
2- The code to find all xlsx files in a folder and read them:
```
l = [] #use a list and concat later, faster than append in the loop
for f in glob.glob("./*.xlsx"):
    df = pd.read_excel(f).reindex(columns=customer_id).dropna(how='all', axis=1)
    df.columns = ["ID"] # to have only one column once concat
    l.append(df)
all_data  = pd.concat(l, ignore_index=True) # concat all data
```
I added the engine openpyxl df = pd.read_excel(f, engine="openpyxl").reindex(columns = customer_id).dropna(how='all', axis=1) Now I got a different error:
```
BadZipFile: File is not a zip file
```
pandas version: 1.3.0 python version: python3.9 os: MacOS is there a better way to read all xlsx files from a folder ?

Response:
Found it. When an excel file is opened for example by MS excel a hidden temporary file is created in the same directory:
```
~$datasheet.xlsx
```
So, when I run the code to read all the files from the folder it gives me the error:
```
Excel file format cannot be determined, you must specify an engine manually.
```
When all files are closed and no hidden temporary files ~$filename.xlsx in the same directory the code works perfectly.


Question:
pandas: complex filter on rows of DataFrame I would like to filter rows by a function of each row, e.g.
```
def f(row):
  return sin(row['velocity'])/np.prod(['masses']) > 5

df = pandas.DataFrame(...)
filtered = df[apply_to_all_rows(df, f)]
```
Or for another more complex, contrived example,
```
def g(row):
  if row['col1'].method1() == 1:
    val = row['col1'].method2() / row['col1'].method3(row['col3'], row['col4'])
  else:
    val = row['col2'].method5(row['col6'])
  return np.sin(val)

df = pandas.DataFrame(...)
filtered = df[apply_to_all_rows(df, g)]
```
How can I do so?

Response:
You can do this using DataFrame.apply, which applies a function along a given axis,
```

```python
df = pandas.DataFrame(np.random.randn(5, 3), columns=['a', 'b', 'c'])
```

```python
df
#Output
#a         b         c
#0 -0.001968 -1.877945 -1.515674
#1 -0.540628  0.793913 -0.983315
#2 -1.313574  1.946410  0.826350
#3  0.015763 -0.267860 -2.228350
#4  0.563111  1.195459  0.343168
```

```python
df[df.apply(lambda x: x['b'] > x['c'], axis=1)]
#Output
#a         b         c
#1 -0.540628  0.793913 -0.983315
#2 -1.313574  1.946410  0.826350
#3  0.015763 -0.267860 -2.228350
#4  0.563111  1.195459  0.343168
#```
```


Question: Plot correlation matrix using pandas I have a data set with huge number of features, so analysing the correlation matrix has become very difficult. I want to plot a correlation matrix which we get using dataframe.corr() function from pandas library. Is there any built-in function provided by the pandas library to plot this matrix?

Response:
If your main goal is to visualize the correlation matrix, rather than creating a plot per se, the convenient pandas styling options is a viable built-in solution:
```
import pandas as pd
import numpy as np

rs = np.random.RandomState(0)
df = pd.DataFrame(rs.rand(10, 10))
corr = df.corr()
corr.style.background_gradient(cmap='coolwarm')
# 'RdBu_r', 'BrBG_r', & PuOr_r are other good diverging colormaps
```
Note that this needs to be in a backend that supports rendering HTML, such as the JupyterLab Notebook. Styling You can easily limit the digit precision (this is now .format(precision=2) in pandas 2.*):
```
corr.style.background_gradient(cmap='coolwarm').set_precision(2)
```
Or get rid of the digits altogether if you prefer the matrix without annotations:
```
corr.style.background_gradient(cmap='coolwarm').set_properties(**{'font-size': '0pt'})
```
The styling documentation also includes instructions of more advanced styles, such as how to change the display of the cell the mouse pointer is hovering over. Time comparison In my testing, style.background_gradient() was 4x faster than plt.matshow() and 120x faster than sns.heatmap() with a 10x10 matrix. Unfortunately it doesn't scale as well as plt.matshow(): the two take about the same time for a 100x100 matrix, and plt.matshow() is 10x faster for a 1000x1000 matrix. Saving There are a few possible ways to save the stylized dataframe: Return the HTML by appending the render() method and then write the output to a file. Save as an .xslx file with conditional formatting by appending the to_excel() method. Combine with imgkit to save a bitmap Take a screenshot (like I have done here). Normalize colors across the entire matrix (pandas >= 0.24) By setting axis=None, it is now possible to compute the colors based on the entire matrix rather than per column or per row:
```
corr.style.background_gradient(cmap='coolwarm', axis=None)
```
Single corner heatmap Since many people are reading this answer I thought I would add a tip for how to only show one corner of the correlation matrix. I find this easier to read myself, since it removes the redundant information.
```
# Fill diagonal and upper half with NaNs
mask = np.zeros_like(corr, dtype=bool)
mask[np.triu_indices_from(mask)] = True
corr[mask] = np.nan
(corr
 .style
 .background_gradient(cmap='coolwarm', axis=None, vmin=-1, vmax=1)
 .highlight_null(color='#f1f1f1')  # Color NaNs grey
 .format(precision=2))
```


Question:
Duplicated rows when merging dataframes in Python I am currently merging two dataframes with an inner join. However, after merging, I see all the rows are duplicated even when the columns that I merged upon contain the same values. Specifically, I have the following code.
```py
merged_df = pd.merge(df1, df2, on=['email_address'], how='inner')
```
Here are the two dataframes and the results. df1
```none
email_address    name   surname
0
[email protected]
    john     smith
1
[email protected]
    john     smith
2
[email protected]
   elvis   presley
```
df2
```none
email_address    street  city
0
[email protected]
   street1    NY
1
[email protected]
   street1    NY
2
[email protected]
   street2    LA
```
merged_df
```none
email_address    name   surname    street  city
0
[email protected]
    john     smith   street1    NY
1
[email protected]
    john     smith   street1    NY
2
[email protected]
    john     smith   street1    NY
3
[email protected]
    john     smith   street1    NY
4
[email protected]
   elvis   presley   street2    LA
5
[email protected]
   elvis   presley   street2    LA
```
My question is, shouldn't it be like this? This is how I would like my merged_df to be like.
```none
email_address    name   surname    street  city
0
[email protected]
    john     smith   street1    NY
1
[email protected]
    john     smith   street1    NY
2
[email protected]
   elvis   presley   street2    LA
```
Are there any ways I can achieve this?

Response:
```
list_2_nodups = list_2.drop_duplicates()
pd.merge(list_1 , list_2_nodups , on=['email_address'])
```
The duplicate rows are expected. Each john smith in list_1 matches with each john smith in list_2. I had to drop the duplicates in one of the lists. I chose list_2.


Question:
Pandas dataframe get first row of each group I have a pandas DataFrame like following:
```
df = pd.DataFrame({'id' : [1,1,1,2,2,3,3,3,3,4,4,5,6,6,6,7,7],
                'value'  : ["first","second","second","first",
                            "second","first","third","fourth",
                            "fifth","second","fifth","first",
                            "first","second","third","fourth","fifth"]})
```
I want to group this by ["id","value"] and get the first row of each group:
```
id   value
0        1   first
1        1  second
2        1  second
3        2   first
4        2  second
5        3   first
6        3   third
7        3  fourth
8        3   fifth
9        4  second
10       4   fifth
11       5   first
12       6   first
13       6  second
14       6   third
15       7  fourth
16       7   fifth
```
Expected outcome:
```
id   value
 1   first
 2   first
 3   first
 4  second
 5  first
 6  first
 7  fourth
```
I tried following, which only gives the first row of the DataFrame.
```

```python
for index, row in df.iterrows():
   ....:     df2 = pd.DataFrame(df.groupby(['id','value']).reset_index().ix[0])
```
```

Response:
Use .first() to get the first (non-null) element.
```

```python
df.groupby('id').first()
#Output
#     value
#id
#1    first
#2    first
#3    first
#4   second
#5    first
#6    first
#7   fourth
#```
#If you need id as column:
#```
```

```python
df.groupby('id').first().reset_index()
#Output
#   id   value
#0   1   first
#1   2   first
#2   3   first
#3   4  second
#4   5   first
#5   6   first
#6   7  fourth
#```
#To get first n records, you can use .head():
#```
```

```python
df.groupby('id').head(2).reset_index(drop=True)
#Output
#    id   value
#0    1   first
#1    1  second
#2    2   first
#3    2  second
#4    3   first
#5    3   third
#6    4  second
#7    4   fifth
#8    5   first
#9    6   first
#10   6  second
#11   7  fourth
#12   7   fifth
#```
```


Question: Plotting pandas timedelta I have a pandas dataframe that has two datetime64 columns and one timedelta64 column that is the difference between the two columns. I'm trying to plot a histogram of the timedelta column to visualize the time differences between the two events. However, just using df['time_delta'] results in: TypeError: ufunc add cannot use operands with types dtype(' df2 = df1['time_delta'].astype(float) results in: TypeError: cannot astype a timedelta from [timedelta64[ns]] to [float64] How would one create a histogram of pandas timedelta data?

Response:
Here are ways to convert timedeltas, docs are here
```

```python
pd.to_timedelta(np.arange(5),unit='d')+pd.to_timedelta(1,unit='s')
#Output
#0   0 days, 00:00:01
#1   1 days, 00:00:01
#2   2 days, 00:00:01
#3   3 days, 00:00:01
#4   4 days, 00:00:01
#dtype: timedelta64[ns]
#```
#Convert to seconds (is an exact conversion)
#```
```

```python
(pd.to_timedelta(np.arange(5),unit='d')+pd.to_timedelta(1,unit='s')).astype('timedelta64[s]')
#Output
#0         1
#1     86401
#2    172801
#3    259201
#4    345601
#dtype: float64
#```
#Convert using astype will round to that unit
#```
```

```python
(pd.to_timedelta(np.arange(5),unit='d')+pd.to_timedelta(1,unit='s')).astype('timedelta64[D]')
#Output
#0    0
#1    1
#2    2
#3    3
#4    4
#dtype: float64
#```
#Division will give an exact repr
#```
```

```python
(pd.to_timedelta(np.arange(5),unit='d')+pd.to_timedelta(1,unit='s')) / np.timedelta64(1,'D')
#Output
#0    0.000012
#1    1.000012
#2    2.000012
#3    3.000012
#4    4.000012
#dtype: float64
#```
```


Question:
How to add an empty column to a dataframe? What's the easiest way to add an empty column to a pandas DataFrame object? The best I've stumbled upon is something like
```py
df['foo'] = df.apply(lambda _: '', axis=1)
```
Is there a less perverse method?

Response:
If I understand correctly, assignment should fill:
```

```python
import numpy as np
```

```python
import pandas as pd
```

```python
df = pd.DataFrame({"A": [1,2,3], "B": [2,3,4]})
```

```python
df
#Output
#   A  B
#0  1  2
#1  2  3
#2  3  4
```

```python
df["C"] = ""
```

```python
df["D"] = np.nan
```

```python
df
#Output
#   A  B C   D
#0  1  2   NaN
#1  2  3   NaN
#2  3  4   NaN
#```
```


Question:
Given a pandas Series that represents frequencies of a value, how can I turn those frequencies into percentages? I was experimenting with the kaggle.com Titanic data set (data on every person on the Titanic) and came up with a gender breakdown like this:
```
df = pd.DataFrame({'sex': ['male'] * 577 + ['female'] * 314})
gender = df.sex.value_counts()
gender

male   577
female 314
```
I would like to find out the percentage of each gender on the Titanic. My approach is slightly less than ideal:
```
from __future__ import division
pcts = gender / gender.sum()
pcts

male      0.647587
female    0.352413
```
Is there a better (more idiomatic) way?

Response:
This function is implemented in pandas, actually even in value_counts(). No need to calculate :) just type:
```
df.sex.value_counts(normalize=True)
```
which gives exactly the desired output. Please note that value_counts() excludes NA values, so numbers might not add up to 1. See here: http://pandas-docs.github.io/pandas-docs-travis/generated/pandas.Series.value_counts.html (A column of a DataFrame is a Series)


Question:
How to convert bytes data into a python pandas dataframe? I would like to convert 'bytes' data into a Pandas dataframe. The data looks like this (few first lines):
```
(b'#Settlement Date,Settlement Period,CCGT,OIL,COAL,NUCLEAR,WIND,PS,NPSHYD,OCGT'
 b',OTHER,INTFR,INTIRL,INTNED,INTEW,BIOMASS\n2017-01-01,1,7727,0,3815,7404,3'
 b'923,0,944,0,2123,948,296,856,238,\n2017-01-01,2,8338,0,3815,7403,3658,16,'
 b'909,0,2124,998,298,874,288,\n2017-01-01,3,7927,0,3801,7408,3925,0,864,0,2'
 b'122,998,298,816,286,\n2017-01-01,4,6996,0,3803,7407,4393,0,863,0,2122,998'
```
The columns headers appear at the top. each subsequent line is a timestamp and numbers. Is there a straightforward way to do this? Thank you very much @Paula Livingstone: This seems to work:
```
s=str(bytes_data,'utf-8')

file = open("data.txt","w")

file.write(s)
df=pd.read_csv('data.txt')
```
maybe this can be done without using a file in between.

Response:
I had the same issue and found this library https://docs.python.org/2/library/stringio.html from the answer here: How to create a Pandas DataFrame from a string Try something like:
```
from io import StringIO

s=str(bytes_data,'utf-8')

data = StringIO(s)

df=pd.read_csv(data)
```


Question:
Convert Pandas Column to DateTime I have one field in a pandas DataFrame that was imported as string format. It should be a datetime variable. How do I convert it to a datetime column, and then filter based on date? Example:
```py
raw_data = pd.DataFrame({'Mycol': ['05SEP2014:00:00:00.000']})
```

Response:
Use the to_datetime function, specifying a format to match your data.
```
df['Mycol'] = pd.to_datetime(df['Mycol'], format='%d%b%Y:%H:%M:%S.%f')
```


Question:
Python: create a pandas data frame from a list I am using the following code to create a data frame from a list:
```
test_list = ['a','b','c','d']
df_test = pd.DataFrame.from_records(test_list, columns=['my_letters'])
df_test
```
The above code works fine. Then I tried the same approach for another list:
```
import pandas as pd
q_list = ['112354401', '116115526', '114909312', '122425491', '131957025', '111373473']
df1 = pd.DataFrame.from_records(q_list, columns=['q_data'])
df1
```
But it gave me the following errors this time:
```
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-24-99e7b8e32a52> in <module>()
      1 import pandas as pd
      2 q_list = ['112354401', '116115526', '114909312', '122425491', '131957025', '111373473']
----> 3 df1 = pd.DataFrame.from_records(q_list, columns=['q_data'])
      4 df1

/usr/local/lib/python3.4/dist-packages/pandas/core/frame.py in from_records(cls, data, index, exclude, columns, coerce_float, nrows)
   1021         else:
   1022             arrays, arr_columns = _to_arrays(data, columns,
-> 1023                                              coerce_float=coerce_float)
   1024
   1025             arr_columns = _ensure_index(arr_columns)

/usr/local/lib/python3.4/dist-packages/pandas/core/frame.py in _to_arrays(data, columns, coerce_float, dtype)
   5550         data = lmap(tuple, data)
   5551         return _list_to_arrays(data, columns, coerce_float=coerce_float,
-> 5552                                dtype=dtype)
   5553
   5554

/usr/local/lib/python3.4/dist-packages/pandas/core/frame.py in _list_to_arrays(data, columns, coerce_float, dtype)
   5607         content = list(lib.to_object_array(data).T)
   5608     return _convert_object_array(content, columns, dtype=dtype,
-> 5609                                  coerce_float=coerce_float)
   5610
   5611

/usr/local/lib/python3.4/dist-packages/pandas/core/frame.py in _convert_object_array(content, columns, coerce_float, dtype)
   5666             # caller's responsibility to check for this...
   5667             raise AssertionError('%d columns passed, passed data had %s '
-> 5668                                  'columns' % (len(columns), len(content)))
   5669
   5670     # provide soft conversion of object dtypes

AssertionError: 1 columns passed, passed data had 9 columns
```
Why would the same approach work for one list but not another? Any idea what might be wrong here? Thanks a lot!

Response:
DataFrame.from_records treats string as a character list. so it needs as many columns as length of string. You could simply use the DataFrame constructor.
```

```python
pd.DataFrame(q_list, columns=['q_data'])
#Output
#q_data
#0  112354401
#1  116115526
#2  114909312
#3  122425491
#4  131957025
#5  111373473
#```
```


Question:
Pandas read_sql with parameters Are there any examples of how to pass parameters with an SQL query in Pandas? In particular I'm using an SQLAlchemy engine to connect to a PostgreSQL database. So far I've found that the following works:
```py
df = psql.read_sql(('select "Timestamp","Value" from "MyTable" '
                     'where "Timestamp" BETWEEN %s AND %s'),
                   db,params=[datetime(2014,6,24,16,0),datetime(2014,6,24,17,0)],
                   index_col=['Timestamp'])
```
The Pandas documentation says that params can also be passed as a dict, but I can't seem to get this to work having tried for instance:
```py
df = psql.read_sql(('select "Timestamp","Value" from "MyTable" '
                     'where "Timestamp" BETWEEN :dstart AND :dfinish'),
                   db,params={"dstart":datetime(2014,6,24,16,0),"dfinish":datetime(2014,6,24,17,0)},
                   index_col=['Timestamp'])
```
What is the recommended way of running these types of queries from Pandas?

Response:
The read_sql docs say this params argument can be a list, tuple or dict (see docs). To pass the values in the sql query, there are different syntaxes possible: ?, :1, :name, %s, %(name)s (see PEP249). But not all of these possibilities are supported by all database drivers, which syntax is supported depends on the driver you are using (psycopg2 in your case I suppose). In your second case, when using a dict, you are using 'named arguments', and according to the psycopg2 documentation, they support the %(name)s style (and so not the :name I suppose), see http://initd.org/psycopg/docs/usage.html#query-parameters. So using that style should work:
```
df = psql.read_sql(('select "Timestamp","Value" from "MyTable" '
                     'where "Timestamp" BETWEEN %(dstart)s AND %(dfinish)s'),
                   db,params={"dstart":datetime(2014,6,24,16,0),"dfinish":datetime(2014,6,24,17,0)},
                   index_col=['Timestamp'])
```


Question:
Appending two dataframes with same columns, different order I have two pandas dataframes.
```
noclickDF = DataFrame([[0, 123, 321], [0, 1543, 432]],
                      columns=['click', 'id', 'location'])
clickDF = DataFrame([[1, 123, 421], [1, 1543, 436]],
                      columns=['click', 'location','id'])
```
I simply want to join such that the final DF will look like:
```
click  |  id   |   location
0         123        321
0         1543       432
1         421        123
1         436       1543
```
As you can see the column names of both original DF's are the same, but not in the same order. Also there is no join in a column.

Response:
You could also use pd.concat:
```

```python
pd.concat([noclickDF, clickDF], ignore_index=True)
#Output
#click    id  location
#0      0   123       321
#1      0  1543       432
#2      1   421       123
#3      1   436      1543
#```
#Under the hood, DataFrame.append calls pd.concat. DataFrame.append has code for handling various types of input, such as Series, tuples, lists and dicts. If you pass it a DataFrame, it passes straight through to pd.concat, so using pd.concat is a bit more direct.
```


Question:
Check which columns in DataFrame are Categorical I want to a simple and generic way to find which columns are categorical in my DataFrame, when I don't manually specify each column type, unlike in this SO question. The df is created with:
```
import pandas as pd
df = pd.read_csv("test.csv", header=None)
```
e.g.
```
0         1         2         3        4
0   1.539240  0.423437 -0.687014   Chicago   Safari
1   0.815336  0.913623  1.800160    Boston   Safari
2   0.821214 -0.824839  0.483724  New York   Safari
```
. UPDATE (2018/02/04) The question assumes numerical columns are NOT categorical, @Zero's accepted answer solves this. **As @Sagarkar's comment points out Programming Data Types and Statistical Data Types (Categorical/Ordinal/Nominal/Numerical) types are orthogonal concepts, thus mapping between them isn't straightforward. @Jeff's answer specifies the precise manner to manually specify the statistical data types to individual df columns.

Response:
You could use df._get_numeric_data() to get numeric columns and then find out categorical columns
```

```python
cols = df.columns
```

```python
num_cols = df._get_numeric_data().columns
```

```python
num_cols
#Output
#Index([u'0', u'1', u'2'], dtype='object')
```

```python
list(set(cols) - set(num_cols))
#Output
#['3', '4']
#```
```


Question:
Output different precision by column with pandas.DataFrame.to_csv()? Question Is it possible to specify a float precision specifically for each column to be printed by the Python pandas package method pandas.DataFrame.to_csv? Background If I have a pandas dataframe that is arranged like this:
```

```python
df_data[:5]
#Output
#year  month  day       lats       lons  vals
#0   2012      6   16  81.862745 -29.834254   0.0
#1   2012      6   16  81.862745 -29.502762   0.1
#2   2012      6   16  81.862745 -29.171271   0.0
#3   2012      6   16  81.862745 -28.839779   0.2
#4   2012      6   16  81.862745 -28.508287   0.0
#```
#There is the float_format option that can be used to specify a precision, but this applys that precision to all columns of the dataframe when printed. When I use that like so:
#```
#df_data.to_csv(outfile, index=False,
#                   header=False, float_format='%11.6f')
#```
#I get the following, where vals is given an inaccurate precision:
#```
#2012,6,16,  81.862745, -29.834254,   0.000000
#2012,6,16,  81.862745, -29.502762,   0.100000
#2012,6,16,  81.862745, -29.171270,   0.000000
#2012,6,16,  81.862745, -28.839779,   0.200000
#2012,6,16,  81.862745, -28.508287,   0.000000
#```
```

Response:
Change the type of column "vals" prior to exporting the data frame to a CSV file
```
df_data['vals'] = df_data['vals'].map(lambda x: '%2.1f' % x)

df_data.to_csv(outfile, index=False, header=False, float_format='%11.6f')
```


Question:
Fill a new pandas column with row numbers I have the following DataFrame data with random index values:
```
A   B
100   0   7
203   5   4
5992  0  10
2003  9   8
20   10   5
12    6   2
```
I would like to add a new column 'C' with row numbers. For example:
```
A   B   C
100   0   7   0
203   5   4   1
5992  0  10   2
2003  9   8   3
20   10   5   4
12    6   2   5
```

Response:
Use numpy.arange or range by length of DataFrame:
```
df['C'] = np.arange(len(df))

df['C'] = range(len(df))
```
Or you can use DataFrame.shape, thank you @Mehmet Burak Sayıcı:
```
df['C'] = np.arange(df.shape[0])
```
```
print (df)
       A   B  C
100    0   7  0
203    5   4  1
5992   0  10  2
2003   9   8  3
20    10   5  4
12     6   2  5
```


Question: How to write DataFrame to postgres table There is DataFrame.to_sql method, but it works only for mysql, sqlite and oracle databases. I cant pass to this method postgres connection or sqlalchemy engine.

Response:
Starting from pandas 0.14 (released end of May 2014), postgresql is supported. The sql module now uses sqlalchemy to support different database flavors. You can pass a sqlalchemy engine for a postgresql database (see docs). E.g.:
```
from sqlalchemy import create_engine
engine = create_engine('postgresql://username:password@localhost:5432/mydatabase')
df.to_sql('table_name', engine)
```
You are correct that in pandas up to version 0.13.1 postgresql was not supported. If you need to use an older version of pandas, here is a patched version of pandas.io.sql: https://gist.github.com/jorisvandenbossche/10841234. I wrote this a time ago, so cannot fully guarantee that it always works, buth the basis should be there). If you put that file in your working directory and import it, then you should be able to do (where con is a postgresql connection):
```
import sql  # the patched version (file is named sql.py)
sql.write_frame(df, 'table_name', con, flavor='postgresql')
```


Question:
How to add a single item to a Pandas Series [duplicate] This question already has answers here: Creating an empty Pandas DataFrame, and then filling it (8 answers) Closed 3 years ago. How do I add a single item to a Pandas Series instance? I'm looking for code along the lines of
```

```python
x = Series()
```

```python
N = 4
```

```python
for i in xrange(N):
```

```python
    x.some_appending_function(i**2)
```

```python
print(x)
#Output
#0 | 0
#1 | 1
#2 | 4
#3 | 9
#```
#Similarly, how can i add a single row to a Pandas DataFrame?
```

Response:
TLDR: do not append items to a series one by one, better extend with an ordered collection I think the question in its current form is a bit tricky. And the accepted answer does answer the question. But the more I use pandas, the more I understand that it's a bad idea to append items to a Series one by one. I'll try to explain why for pandas beginners. You might think that appending data to a given Series might allow you to reuse some resources, but in reality a Series is just a container that stores a relation between an index and a values array. Each is a numpy.array under the hood, and the index is immutable. When you add to Series an item with a label that is missing in the index, a new index with size n+1 is created, and a new values values array of the same size. That means that when you append items one by one, you create two more arrays of the n+1 size on each step. By the way, you can not append a new item by position (you will get an IndexError) and the label in an index does not have to be unique, that is when you assign a value with a label, you assign the value to all existing items with the the label, and a new row is not appended in this case. This might lead to subtle bugs. The moral of the story is that you should not append data one by one, you should better extend with an ordered collection. The problem is that you can not extend a Series inplace. That is why it is better to organize your code so that you don't need to update a specific instance of a Series by reference. If you create labels yourself and they are increasing, the easiest way is to add new items to a dictionary, then create a new Series from the dictionary (it sorts the keys) and append the Series to an old one. If the keys are not increasing, then you will need to create two separate lists for the new labels and the new values. Below are some code samples:
```

```python
import pandas as pd
```

```python
import numpy as np
```

```python
s = pd.Series(np.arange(4)**2, index=np.arange(4))
```

```python
s
#Output
#0    0
#1    1
#2    4
#3    9
#dtype: int64
```

```python
id(s.index), id(s.values)
#Output
#(4470549648, 4470593296)
#```
#When we update an existing item, the index and the values array stay the same (if you do not change the type of the value)
#```
```

```python
s[2] = 14
```

```python
id(s.index), id(s.values)
#Output
#(4470549648, 4470593296)
#```
#But when you add a new item, a new index and a new values array is generated:
#```
```

```python
s[4] = 16
```

```python
s
#Output
#0     0
#1     1
#2    14
#3     9
#4    16
#dtype: int64
```

```python
id(s.index), id(s.values)
#Output
#(4470548560, 4470595056)
#```
#That is if you are going to append several items, collect them in a dictionary, create a Series, append it to the old one and save the result:
#```
```

```python
new_items = {item: item**2 for item in range(5, 7)}
```

```python
s2 = pd.Series(new_items)
```

```python
s2  # keys are guaranteed to be sorted!
#Output
#5    25
#6    36
#dtype: int64
```

```python
s = s.append(s2); s
#Output
#0     0
#1     1
#2    14
#3     9
#4    16
#5    25
#6    36
#dtype: int64
#```
```


Question:
FutureWarning: The default value of regex will change from True to False in a future version I'm running below code to clean text
```
import pandas as pd

def not_regex(pattern):
        return r"((?!{}).)".format(pattern)

tmp = pd.DataFrame(['No one has a European accent either @',
                    'That the kid   reminds me of Kevin'])

tmp[0].str.replace(not_regex('(\\b[-/]\\b|[a-zA-Z0-9])'), ' ')
```
Then it returns a warning
```
<ipython-input-8-ef8a43f91dbd>:9: FutureWarning: The default value of regex will change from True to False in a future version.
  tmp[0].str.replace(not_regex('(\\b[-/]\\b|[a-zA-Z0-9])'), ' ')
```
Could you please elaborate on the reason of this warning?

Response:
See Pandas 1.2.0 release notes: The default value of regex for Series.str.replace() will change from True to False in a future release. In addition, single character regular expressions will not be treated as literal strings when regex=True is set (GH24804) I.e., use regular expressions explicitly now:
```py
dframe['colname'] = dframe['colname'].str.replace(r'\D+', regex=True)
```


Question:
Create an empty data frame with index from another data frame I've got a data frame df1 with multiple columns and rows. Simple example:
```
TIME T1  T2
       1 10 100
       2 20 200
       3 30 300
```
I'd like to create an empty data frame df2 and later on, add new columns with the calculation results. For this moment my code looks like this:
```
df1=pd.read_csv("1.txt",index_col="TIME")

     df2=df1.copy()[[]] #copy df1 and erase all columns
```
...adding two new columns:
```
df2["results1"],df2["results2"]=df1["T1"]*df["T2"]*3,df1["T2"]+100
```
Is there any better/safer/faster way to do this ? Is it possible to create an empty data frame df2 and only copy index from df1 ?

Response:
```
df2 = pd.DataFrame(index=df1.index)
```
This will create a DataFrame with no columns but just an index, and it will be the same index as in the df1.


Question: How do I filter a pandas DataFrame based on value counts? I'm working in Python with a pandas DataFrame of video games, each with a genre. I'm trying to remove any video game with a genre that appears less than some number of times in the DataFrame, but I have no clue how to go about this. I did find a StackOverflow question that seems to be related, but I can't decipher the solution at all (possibly because I've never heard of R and my memory of functional programming is rusty at best). Help?

Response:
Use groupby filter:
```

```python
df = pd.DataFrame([[1, 2], [1, 4], [5, 6]], columns=['A', 'B'])
```

```python
df
#Output
#A  B
#0  1  2
#1  1  4
#2  5  6
```

```python
df.groupby("A").filter(lambda x: len(x) > 1)
#Output
#A  B
#0  1  2
#1  1  4
#```
#I recommend reading the split-combine-section of the docs.
```


Question: How do I convert timestamp to datetime.date in pandas dataframe? I need to merge 2 pandas dataframes together on dates, but they currently have different date types. 1 is timestamp (imported from excel) and the other is datetime.date. Any advice? I've tried pd.to_datetime().date but this only works on a single item(e.g. df.ix[0,0]), it won't let me apply to the entire series (e.g. df['mydates']) or the dataframe.

Response: I got some help from a colleague. This appears to solve the problem posted above pd.to_datetime(df['mydates']).apply(lambda x: x.date())


Question:
Pandas dataframe select rows where a list-column contains any of a list of strings I've got a pandas DataFrame that looks like this:
```
molecule            species
0        a              [dog]
1        b       [horse, pig]
2        c         [cat, dog]
3        d  [cat, horse, pig]
4        e     [chicken, pig]
```
and I like to extract a DataFrame containing only thoses rows, that contain any of selection = ['cat', 'dog']. So the result should look like this:
```
molecule            species
0        a              [dog]
1        c         [cat, dog]
2        d  [cat, horse, pig]
```
What would be the simplest way to do this? For testing:
```
selection = ['cat', 'dog']
df = pd.DataFrame({'molecule': ['a','b','c','d','e'], 'species' : [['dog'], ['horse','pig'],['cat', 'dog'], ['cat','horse','pig'], ['chicken','pig']]})
```

Response:
IIUC Re-create your df then using isin with any should be faster than apply
```
df[pd.DataFrame(df.species.tolist()).isin(selection).any(1).values]
Out[64]:
  molecule            species
0        a              [dog]
2        c         [cat, dog]
3        d  [cat, horse, pig]
```


Question:
Why use loc in Pandas? Why do we use loc for pandas dataframes? it seems the following code with or without using loc both compiles and runs at a similar speed:
```
%timeit df_user1 = df.loc[df.user_id=='5561']

100 loops, best of 3: 11.9 ms per loop
```
or
```
%timeit df_user1_noloc = df[df.user_id=='5561']

100 loops, best of 3: 12 ms per loop
```
So why use loc? Edit: This has been flagged as a duplicate question. But although pandas iloc vs ix vs loc explanation? does mention that you can do column retrieval just by using the data frame's __getitem__:
```
df['time']    # equivalent to df.loc[:, 'time']
```
it does not say why we use loc, although it does explain lots of features of loc. But my specific question is: why not just omit loc altogether? For this question, I have accepted a very detailed answer below. Also in the above post, the answer (which I do not think is an answer) is really well hidden in the discussion, and any person searching for what I was, would find it hard to locate the information and would be much better served by the answer provided to my question here.

Response:
Explicit is better than implicit. df[boolean_mask] selects rows where boolean_mask is True, but there is a corner case when you might not want it to: when df has boolean-valued column labels:
```

```python
df = pd.DataFrame({True:[1,2,3],False:[3,4,5]}); df
#Output
#False  True
#0      3      1
#1      4      2
#2      5      3
#```
#You might want to use df[[True]] to select the True column. Instead it raises a ValueError:
#```
```

```python
df[[True]]
ValueError: Item wrong length 1 instead of 3.
```
Versus using loc:
```
```

```python
df.loc[[True]]
#Output
#False  True
#0      3      1
#```
#In contrast, the following does not raise ValueError even though the structure of df2 is almost the same as df1 above:
#```
```

```python
df2 = pd.DataFrame({'A':[1,2,3],'B':[3,4,5]}); df2
#Output
#A  B
#0  1  3
#1  2  4
#2  3  5
```

```python
df2[['B']]
#Output
#B
#0  3
#1  4
#2  5
#```
#Thus, df[boolean_mask] does not always behave the same as df.loc[boolean_mask]. Even though this is arguably an unlikely use case, I would recommend always using df.loc[boolean_mask] instead of df[boolean_mask] because the meaning of df.loc's syntax is explicit. With df.loc[indexer] you know automatically that df.loc is selecting rows. In contrast, it is not clear if df[indexer] will select rows or columns (or raise ValueError) without knowing details about indexer and df. df.loc[row_indexer, column_index] can select rows and columns. df[indexer] can only select rows or columns depending on the type of values in indexer and the type of column values df has (again, are they boolean?).
#```
```

```python
df2.loc[[True,False,True], 'B']
#Output
#0    3
#2    5
#Name: B, dtype: int64
#```
#When a slice is passed to df.loc the end-points are included in the range. When a slice is passed to df[...], the slice is interpreted as a half-open interval:
#```
```

```python
df2.loc[1:2]
#Output
#A  B
#1  2  4
#2  3  5
```

```python
df2[1:2]
#Output
#A  B
#1  2  4
#```
```


Question:
How to get the first column of a pandas DataFrame as a Series? I tried:
```
x=pandas.DataFrame(...)
s = x.take([0], axis=1)
```
And s gets a DataFrame, not a Series.

Response:
```

```python
import pandas as pd
```

```python
df = pd.DataFrame({'x' : [1, 2, 3, 4], 'y' : [4, 5, 6, 7]})
```

```python
df
#Output
#   x  y
#0  1  4
#1  2  5
#2  3  6
#3  4  7
```

```python
s = df.ix[:,0]
```

```python
type(s)
#Output
#<class 'pandas.core.series.Series'>
#>>>
#```
#=========================================================================== UPDATE If you're reading this after June 2017, ix has been deprecated in pandas 0.20.2, so don't use it. Use loc or iloc instead. See comments and other answers to this question.
```


Question:
How to convert SQL Query result to PANDAS Data Structure? Any help on this problem will be greatly appreciated. So basically I want to run a query to my SQL database and store the returned data as Pandas data structure. I have attached code for query. I am reading the documentation on Pandas, but I have problem to identify the return type of my query. I tried to print the query result, but it doesn't give any useful information. Thanks!!!!
```py
from sqlalchemy import create_engine

engine2 = create_engine('mysql://THE DATABASE I AM ACCESSING')
connection2 = engine2.connect()
dataid = 1022
resoverall = connection2.execute("
    SELECT
       sum(BLABLA) AS BLA,
       sum(BLABLABLA2) AS BLABLABLA2,
       sum(SOME_INT) AS SOME_INT,
       sum(SOME_INT2) AS SOME_INT2,
       100*sum(SOME_INT2)/sum(SOME_INT) AS ctr,
       sum(SOME_INT2)/sum(SOME_INT) AS cpc
    FROM daily_report_cooked
    WHERE campaign_id = '%s'",
    %dataid
)
```
So I sort of want to understand what's the format/datatype of my variable "resoverall" and how to put it with PANDAS data structure.

Response:
Edit: Mar. 2015 As noted below, pandas now uses SQLAlchemy to both read from (read_sql) and insert into (to_sql) a database. The following should work
```py
import pandas as pd

df = pd.read_sql(sql, cnxn)
```
Previous answer: Via mikebmassey from a similar question
```py
import pyodbc
import pandas.io.sql as psql

cnxn = pyodbc.connect(connection_info)
cursor = cnxn.cursor()
sql = "SELECT * FROM TABLE"

df = psql.frame_query(sql, cnxn)
cnxn.close()
```


Question:
add title to collection of pandas hist plots I'm looking for advice on how to show a title at the top of a collection of histogram plots that have been generated by a pandas df.hist() command. For instance, in the histogram figure block generated by the code below I'd like to place a general title (e.g. 'My collection of histogram plots') at the top of the figure:
```
data = DataFrame(np.random.randn(500).reshape(100,5), columns=list('abcde'))
axes = data.hist(sharey=True, sharex=True)
```
I've tried using the title keyword in the hist command (i.e. title='My collection of histogram plots'), but that didn't work. The following code does work (in an ipython notebook) by adding text to one of the axes, but is a bit of a kludge.
```
axes[0,1].text(0.5, 1.4,'My collection of histogram plots', horizontalalignment='center',
               verticalalignment='center', transform=axes[0,1].transAxes)
```
Is there a better way?

Response:
With newer Pandas versions, if someone is interested, here a slightly different solution with Pandas only:
```
ax = data.plot(kind='hist',subplots=True,sharex=True,sharey=True,title='My title')
```


Question:
Remap values in pandas column with a dict, preserve NaNs I have a dictionary which looks like this: di = {1: "A", 2: "B"} I would like to apply it to the col1 column of a dataframe similar to:
```none
col1   col2
0       w      a
1       1      2
2       2    NaN
```
to get:
```none
col1   col2
0       w      a
1       A      2
2       B    NaN
```
How can I best do this?

Response:
map can be much faster than replace If your dictionary has more than a couple of keys, using map can be much faster than replace. There are two versions of this approach, depending on whether your dictionary exhaustively maps all possible values (and also whether you want non-matches to keep their values or be converted to NaNs): Exhaustive Mapping In this case, the form is very simple:
```
df['col1'].map(di)       # note: if the dictionary does not exhaustively map all
                         # entries then non-matched entries are changed to NaNs
```
Although map most commonly takes a function as its argument, it can alternatively take a dictionary or series: Documentation for Pandas.series.map Non-Exhaustive Mapping If you have a non-exhaustive mapping and wish to retain the existing variables for non-matches, you can add fillna:
```
df['col1'].map(di).fillna(df['col1'])
```
as in @jpp's answer here: Replace values in a pandas series via dictionary efficiently Benchmarks Using the following data with pandas version 0.23.1:
```
di = {1: "A", 2: "B", 3: "C", 4: "D", 5: "E", 6: "F", 7: "G", 8: "H" }
df = pd.DataFrame({ 'col1': np.random.choice( range(1,9), 100000 ) })
```
and testing with %timeit, it appears that map is approximately 10x faster than replace. Note that your speedup with map will vary with your data. The largest speedup appears to be with large dictionaries and exhaustive replaces. See @jpp answer (linked above) for more extensive benchmarks and discussion.


Question:
Compare two DataFrames and output their differences side-by-side I am trying to highlight exactly what changed between two dataframes. Suppose I have two Python Pandas dataframes:
```
"StudentRoster Jan-1":
id   Name   score                    isEnrolled           Comment
111  Jack   2.17                     True                 He was late to class
112  Nick   1.11                     False                Graduated
113  Zoe    4.12                     True

"StudentRoster Jan-2":
id   Name   score                    isEnrolled           Comment
111  Jack   2.17                     True                 He was late to class
112  Nick   1.21                     False                Graduated
113  Zoe    4.12                     False                On vacation
```
My goal is to output an HTML table that: Identifies rows that have changed (could be int, float, boolean, string) Outputs rows with same, OLD and NEW values (ideally into an HTML table) so the consumer can clearly see what changed between two dataframes:
```
"StudentRoster Difference Jan-1 - Jan-2":
id   Name   score                    isEnrolled           Comment
112  Nick   was 1.11| now 1.21       False                Graduated
113  Zoe    4.12                     was True | now False was "" | now   "On   vacation"
```
I suppose I could do a row by row and column by column comparison, but is there an easier way?

Response:
The first part is similar to Constantine, you can get the boolean of which rows are empty*:
```

```python
ne = (df1 != df2).any(1)
```

```python
ne
#Output
#0    False
#1     True
#2     True
#dtype: bool
#```
#Then we can see which entries have changed:
#```
```

```python
ne_stacked = (df1 != df2).stack()
```

```python
changed = ne_stacked[ne_stacked]
```

```python
changed.index.names = ['id', 'col']
```

```python
changed
#Output
#id  col
#1   score         True
#2   isEnrolled    True
#    Comment       True
#dtype: bool
#```
#Here the first entry is the index and the second the columns which has been changed.
#```
```

```python
difference_locations = np.where(df1 != df2)
```

```python
changed_from = df1.values[difference_locations]
```

```python
changed_to = df2.values[difference_locations]
```

```python
pd.DataFrame({'from': changed_from, 'to': changed_to}, index=changed.index)
#Output
#from           to
#id col
#1  score       1.11         1.21
#2  isEnrolled  True        False
#   Comment     None  On vacation
#```
#* Note: it's important that df1 and df2 share the same index here. To overcome this ambiguity, you can ensure you only look at the shared labels using df1.index & df2.index, but I think I'll leave that as an exercise.
```


Question:
how to replace an entire column on Pandas.DataFrame I would like to replace an entire column on a Pandas DataFrame with another column taken from another DataFrame, an example will clarify what I am looking for
```
import pandas as pd
dic = {'A': [1, 4, 1, 4], 'B': [9, 2, 5, 3], 'C': [0, 0, 5, 3]}
df = pd.DataFrame(dic)
```
df is
```
'A' 'B' 'C'
 1   9   0
 4   2   0
 1   5   5
 4   3   3
```
Now I have another dataframe called df1 with a column "E" that is
```
df1['E'] = [ 4, 4, 4, 0]
```
and I would like to replace column "B" of df with column "E" of df1
```
'A' 'E' 'C'
 1   4   0
 4   4   0
 1   4   5
 4   0   3
```
I tried to use the .replace() method in many ways but I didn't get anything good. Can you help me?

Response:
If the indices match then:
```
df['B'] = df1['E']
```
should work otherwise:
```
df['B'] = df1['E'].values
```
will work so long as the length of the elements matches


Question:
How to copy/paste DataFrame from Stack Overflow into Python In questions and answers, users very often post an example DataFrame which their question/answer works with:
```

```python
x
#Output
#bar  foo
#0    4    1
#1    5    2
#2    6    3
#```
#It'd be really useful to be able to get this DataFrame into my Python interpreter so I can start debugging the question, or testing the answer. How can I do this?
```

Response:
Pandas is written by people that really know what people want to do. Since version 0.13 there's a function pd.read_clipboard which is absurdly effective at making this "just work". Copy and paste the part of the code in the question that starts bar foo, (i.e. the DataFrame) and do this in a Python interpreter:
```

```python
import pandas as pd
```

```python
df = pd.read_clipboard()
```

```python
df
#Output
#bar  foo
#0    4    1
#1    5    2
#2    6    3
#```
#Caveats Don't include the iPython In or Out stuff or it won't work If you have a named index, you currently need to add engine='python' (see this issue on GitHub). The 'c' engine is currently broken when the index is named. It's not brilliant at MultiIndexes: Try this:
#```
#0         1         2
#level1 level2
#foo    a       0.518444  0.239354  0.364764
#       b       0.377863  0.912586  0.760612
#bar    a       0.086825  0.118280  0.592211
#```
#which doesn't work at all, or this:
#```
#0         1         2
#foo a  0.859630  0.399901  0.052504
#    b  0.231838  0.863228  0.017451
#bar a  0.422231  0.307960  0.801993
#```
#Which works, but returns something totally incorrect!
```


Question:
Pandas get topmost n records within each group Suppose I have pandas DataFrame like this:
```py
df = pd.DataFrame({'id':[1,1,1,2,2,2,2,3,4], 'value':[1,2,3,1,2,3,4,1,1]})
```
which looks like:
```none
id  value
0   1      1
1   1      2
2   1      3
3   2      1
4   2      2
5   2      3
6   2      4
7   3      1
8   4      1
```
I want to get a new DataFrame with top 2 records for each id, like this:
```none
id  value
0   1      1
1   1      2
3   2      1
4   2      2
7   3      1
8   4      1
```
I can do it with numbering records within group after groupby:
```py
dfN = df.groupby('id').apply(lambda x:x['value'].reset_index()).reset_index()
```
which looks like:
```none
id  level_1  index  value
0   1        0      0      1
1   1        1      1      2
2   1        2      2      3
3   2        0      3      1
4   2        1      4      2
5   2        2      5      3
6   2        3      6      4
7   3        0      7      1
8   4        0      8      1
```
then for the desired output:
```py
dfN[dfN['level_1'] <= 1][['id', 'value']]
```
Output:
```none
id  value
0   1      1
1   1      2
3   2      1
4   2      2
7   3      1
8   4      1
```
But is there more effective/elegant approach to do this? And also is there more elegant approach to number records within each group (like SQL window function row_number()).

Response:
Did you try
```
df.groupby('id').head(2)
```
Output generated:
```
id  value
id
1  0   1      1
   1   1      2
2  3   2      1
   4   2      2
3  7   3      1
4  8   4      1
```
(Keep in mind that you might need to order/sort before, depending on your data) EDIT: As mentioned by the questioner, use
```
df.groupby('id').head(2).reset_index(drop=True)
```
to remove the MultiIndex and flatten the results:
```
id  value
0   1      1
1   1      2
2   2      1
3   2      2
4   3      1
5   4      1
```


Question: Large, persistent DataFrame in pandas I am exploring switching to python and pandas as a long-time SAS user. However, when running some tests today, I was surprised that python ran out of memory when trying to pandas.read_csv() a 128mb csv file. It had about 200,000 rows and 200 columns of mostly numeric data. With SAS, I can import a csv file into a SAS dataset and it can be as large as my hard drive. Is there something analogous in pandas? I regularly work with large files and do not have access to a distributed computing network.

Response:
Wes is of course right! I'm just chiming in to provide a little more complete example code. I had the same issue with a 129 Mb file, which was solved by:
```
import pandas as pd

# Returns a TextFileReader, which is iterable with chunks of 1000 rows.
csv_iterator = pd.read_csv('large_dataset.csv', iterator=True, chunksize=1000)
# Iterate through the dataframe chunks and print one row/record at a time
for chunk in csv_iterator:
    for index, row in chunk.iterrows():
        print(row)

# df is DataFrame. If errors, use `list(csv_iterator)` instead
df = pd.concat(tp, ignore_index=True)
```


Question:
How to turn a pandas dataframe row into a comma separated string I need to iterate over each row of a pandas df and turn this into a comma separated string. example:
```
df3 = DataFrame(np.random.randn(10, 5),
              columns=['a', 'b', 'c', 'd', 'e'])

          a         b         c         d         e
0 -0.158897 -0.749799  0.268921  0.070035  0.099600
1 -0.863654 -0.086814 -0.614562 -1.678850  0.980292
2 -0.098168  0.710652 -0.456274 -0.373153 -0.533463
3  1.001634 -0.736187 -0.812034  0.223062 -1.337972
4  0.173549 -0.576412 -1.016063 -0.217242  0.443794
5  0.273695  0.335562  0.778393 -0.668368  0.438880
6 -0.783824  1.439888  1.057639 -1.825481 -0.770953
7 -1.025004  0.155974  0.645023  0.993379 -0.812133
8  0.953448 -1.355628 -1.918317 -0.966472 -0.618744
9 -0.479297  0.295150 -0.294449  0.679416 -1.813078
```
I'd like to get for each row:
```
'-0.158897,-0.749799,0.268921,0.070035,0.099600'
 '0.863654,-0.086814,-0.614562,-1.678850,0.980292'
... and so on
```

Response:
You could use pandas.DataFrame.to_string with some optional arguments set to False and then split on newline characters to get a list of your strings. This feels a little dirty though.
```
x = df3.to_string(header=False,
                  index=False,
                  index_names=False).split('\n')
vals = [','.join(ele.split()) for ele in x]
print(vals)
```
Outputs:
```
['1.221365,0.923175,-1.286149,-0.153414,-0.005078', '-0.231824,-1.131186,0.853728,0.160349,1.000170', '-0.147145,0.310587,-0.388535,0.957730,-0.185315', '-1.658463,-1.114204,0.760424,-1.504126,0.206909', '-0.734571,0.908569,-0.698583,-0.692417,-0.768087', '0.000029,0.204140,-0.483123,-1.064851,-0.835931', '-0.108869,0.426260,0.107286,-1.184402,0.434607', '-0.692160,-0.376433,0.567188,-0.171867,-0.822502', '-0.564726,-1.084698,-1.065283,-2.335092,-0.083357', '-1.429049,0.790535,-0.547701,-0.684346,2.048081']
```


Question: What is the difference between save a pandas dataframe to pickle and to csv? I am learning python pandas. I see a tutorial which shows two ways to save a pandas dataframe. pd.to_csv('sub.csv') and to open pd.read_csv('sub.csv') pd.to_pickle('sub.pkl') and to open pd.read_pickle('sub.pkl') The tutorial says to_pickle is to save the dataframe to disk. I am confused about this. Because when I use to_csv, I did see a csv file appears in the folder, which I assume is also save to disk right? In general, why we want to save a dataframe using to_pickle rather than save it to csv or txt or other format?

Response: csv ✅human readable ✅cross platform ⛔slower ⛔more disk space ⛔doesn't preserve types in some cases pickle ✅fast saving/loading ✅less disk space ⛔non human readable ⛔python only ⛔sensitive to version changes Also take a look at parquet format (to_parquet, read_parquet) ✅fast saving/loading ✅less disk space than pickle ✅supported by many platforms ⛔non human readable


Question:
Any way to get mappings of a label encoder in Python pandas? I am converting strings to categorical values in my dataset using the following piece of code.
```
data['weekday'] = pd.Categorical.from_array(data.weekday).labels
```
For eg,
```
index    weekday
0        Sunday
1        Sunday
2        Wednesday
3        Monday
4        Monday
5        Thursday
6        Tuesday
```
After encoding the weekday, my dataset appears like this:
```
index    weekday
    0       3
    1       3
    2       6
    3       1
    4       1
    5       4
    6       5
```
Is there any way I can know that Sunday has been mapped to 3, Wednesday to 6 and so on?

Response:
You can create additional dictionary with mapping:
```
from sklearn import preprocessing
le = preprocessing.LabelEncoder()
le.fit(data['name'])
le_name_mapping = dict(zip(le.classes_, le.transform(le.classes_)))
print(le_name_mapping)
{'Tom': 0, 'Nick': 1, 'Kate': 2}
```


Question:
Convert multiple columns to string in pandas dataframe I have a pandas data frame with different data types. I want to convert more than one column in the data frame to string type. I have individually done for each column but want to know if there is an efficient way? So at present I am doing something like this:
```
repair['SCENARIO']=repair['SCENARIO'].astype(str)

repair['SERVICE_TYPE']= repair['SERVICE_TYPE'].astype(str)
```
I want a function that would help me pass multiple columns and convert them to strings.

Response:
To convert multiple columns to string, include a list of columns to your above-mentioned command:
```
df[['one', 'two', 'three']] = df[['one', 'two', 'three']].astype(str)
# add as many column names as you like.
```
That means that one way to convert all columns is to construct the list of columns like this:
```
all_columns = list(df) # Creates list of all column headers
df[all_columns] = df[all_columns].astype(str)
```
Note that the latter can also be done directly (see comments).


Question:
Cumulative sum and percentage on column? I have a DataFrame like this: df:
```
fruit    val1 val2
0 orange    15    3
1 apple     10   13
2 mango     5    5
```
How do I get Pandas to give me a cumulative sum and percentage column on only val1? Desired output: df_with_cumsum:
```
fruit    val1 val2   cum_sum    cum_perc
0 orange    15    3    15          50.00
1 apple     10   13    25          83.33
2 mango     5    5     30          100.00
```
I tried df.cumsum(), but it's giving me this error: TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Response:
```
df['cum_sum'] = df['val1'].cumsum()
df['cum_perc'] = 100*df['cum_sum']/df['val1'].sum()
```
This will add the columns to df. If you want a copy, copy df first and then do these operations on the copy.


Question:
pandas add column to groupby dataframe I have this simple dataframe df:
```
df = pd.DataFrame({'c':[1,1,1,2,2,2,2],'type':['m','n','o','m','m','n','n']})
```
my goal is to count values of type for each c, and then add a column with the size of c. So starting with:
```

```python
g = df.groupby('c')['type'].value_counts().reset_index(name='t')
```

```python
g
#Output
#c type  t
#0  1    m  1
#1  1    n  1
#2  1    o  1
#3  2    m  2
#4  2    n  2
#```
#the first problem is solved. Then I can also:
#```
```

```python
a = df.groupby('c').size().reset_index(name='size')
```

```python
a
#Output
#c  size
#0  1     3
#1  2     4
#```
#How can I add the size column directly to the first dataframe? So far I used map as:
#```
```

```python
a.index = a['c']
```

```python
g['size'] = g['c'].map(a['size'])
```

```python
g
#Output
#c type  t  size
#0  1    m  1     3
#1  1    n  1     3
#2  1    o  1     3
#3  2    m  2     4
#4  2    n  2     4
#```
#which works, but is there a more straightforward way to do this?
```

Response:
Use transform to add a column back to the orig df from a groupby aggregation, transform returns a Series with its index aligned to the orig df:
```

```python
g = df.groupby('c')['type'].value_counts().reset_index(name='t')
g['size'] = df.groupby('c')['type'].transform('size')
g
#Output
#c type  t  size
#0  1    m  1     3
#1  1    n  1     3
#2  1    o  1     3
#3  2    m  2     4
#4  2    n  2     4
#```
```


Question:
How to change the color of a single bar in a bar plot Supposely, I have the bar chart as below: Any ideas on how to set different colors for each carrier? As for example, AK would be Red, GA would be Green, etc? I am using Pandas and matplotlib in Python
```

```python
f=plt.figure()
```

```python
ax=f.add_subplot(1,1,1)
```

```python
ax.bar([1,2,3,4], [1,2,3,4])
#Output
#<Container object of 4 artists>
```

```python
ax.get_children()
#Output
#[<matplotlib.axis.XAxis object at 0x6529850>, <matplotlib.axis.YAxis object at 0x78460d0>,  <matplotlib.patches.Rectangle object at 0x733cc50>, <matplotlib.patches.Rectangle object at 0x733cdd0>, <matplotlib.patches.Rectangle object at 0x777f290>, <matplotlib.patches.Rectangle object at 0x777f710>, <matplotlib.text.Text object at 0x7836450>, <matplotlib.patches.Rectangle object at 0x7836390>, <matplotlib.spines.Spine object at 0x6529950>, <matplotlib.spines.Spine object at 0x69aef50>, <matplotlib.spines.Spine object at 0x69ae310>, <matplotlib.spines.Spine object at 0x69aea50>]
```

```python
ax.get_children()[2].set_color('r') #You can also try to locate the first patches.Rectangle object instead of direct calling the index.
#Output
#```
#For the suggestions above, how do exactly we could enumerate ax.get_children() and check if the object type is rectangle? So if the object is rectangle, we would assign different random color?
```

Response:
Simple, just use .set_color
```

```python
barlist=plt.bar([1,2,3,4], [1,2,3,4])
```

```python
barlist[0].set_color('r')
```

```python
plt.show()
#Output
#```
#For your new question, not much harder either, just need to find the bar from your axis, an example:
#```
```

```python
f=plt.figure()
```

```python
ax=f.add_subplot(1,1,1)
```

```python
ax.bar([1,2,3,4], [1,2,3,4])
#Output
#<Container object of 4 artists>
```

```python
ax.get_children()
#Output
#[<matplotlib.axis.XAxis object at 0x6529850>,
# <matplotlib.axis.YAxis object at 0x78460d0>,
# <matplotlib.patches.Rectangle object at 0x733cc50>,
# <matplotlib.patches.Rectangle object at 0x733cdd0>,
# <matplotlib.patches.Rectangle object at 0x777f290>,
# <matplotlib.patches.Rectangle object at 0x777f710>,
# <matplotlib.text.Text object at 0x7836450>,
# <matplotlib.patches.Rectangle object at 0x7836390>,
# <matplotlib.spines.Spine object at 0x6529950>,
# <matplotlib.spines.Spine object at 0x69aef50>,
# <matplotlib.spines.Spine object at 0x69ae310>,
# <matplotlib.spines.Spine object at 0x69aea50>]
```

```python
ax.get_children()[2].set_color('r')
#Output
# #You can also try to locate the first patches.Rectangle object
# #instead of direct calling the index.
#```
#If you have a complex plot and want to identify the bars first, add those:
#```
```

```python
import matplotlib
```

```python
childrenLS=ax.get_children()
```

```python
barlist=filter(lambda x: isinstance(x, matplotlib.patches.Rectangle), childrenLS)
#Output
#[<matplotlib.patches.Rectangle object at 0x3103650>,
# <matplotlib.patches.Rectangle object at 0x3103810>,
# <matplotlib.patches.Rectangle object at 0x3129850>,
# <matplotlib.patches.Rectangle object at 0x3129cd0>,
# <matplotlib.patches.Rectangle object at 0x3112ad0>]
#```
```


Question:
How to get a value from a Pandas DataFrame and not the index and object type Say I have the following DataFrame
```
Letter    Number
A          1
B          2
C          3
D          4
```
Which can be obtained through the following code
```
import pandas as pd

letters = pd.Series(('A', 'B', 'C', 'D'))
numbers = pd.Series((1, 2, 3, 4))
keys = ('Letters', 'Numbers')
df = pd.concat((letters, numbers), axis=1, keys=keys)
```
Now I want to get the value C from the column Letters. The command line
```
df[df.Letters=='C'].Letters
```
will return
```
2    C
Name: Letters, dtype: object
```
How can I get only the value C and not the whole two line output?

Response:
```
df[df.Letters=='C'].Letters.item()
```
This returns the first element in the Index/Series returned from that selection. In this case, the value is always the first element. EDIT: Or you can run a loc() and access the first element that way. This was shorter and is the way I have implemented it in the past. Pandas Index doc Pandas Series doc


Question:
Understanding inplace=True in pandas In the pandas library many times there is an option to change the object inplace such as with the following statement...
```
df.dropna(axis='index', how='all', inplace=True)
```
I am curious what is being returned as well as how the object is handled when inplace=True is passed vs. when inplace=False. Are all operations modifying self when inplace=True? And when inplace=False is a new object created immediately such as new_df = self and then new_df is returned? If you are trying to close a question where someone should use inplace=True and hasn't, consider replace() method not working on Pandas DataFrame instead.

Response:
In pandas, is inplace = True considered harmful, or not? TLDR; Yes, yes it is. inplace, contrary to what the name implies, often does not prevent copies from being created, and (almost) never offers any performance benefits inplace does not work with method chaining inplace can lead to SettingWithCopyWarning if used on a DataFrame column, and may prevent the operation from going though, leading to hard-to-debug errors in code The pain points above are common pitfalls for beginners, so removing this option will simplify the API. I don't advise setting this parameter as it serves little purpose. See this GitHub issue which proposes the inplace argument be deprecated api-wide. It is a common misconception that using inplace=True will lead to more efficient or optimized code. In reality, there are absolutely no performance benefits to using inplace=True. Both the in-place and out-of-place versions create a copy of the data anyway, with the in-place version automatically assigning the copy back. inplace=True is a common pitfall for beginners. For example, it can trigger the SettingWithCopyWarning:
```
df = pd.DataFrame({'a': [3, 2, 1], 'b': ['x', 'y', 'z']})

df2 = df[df['a'] > 1]
df2['b'].replace({'x': 'abc'}, inplace=True)
# SettingWithCopyWarning:
# A value is trying to be set on a copy of a slice from a DataFrame
```
Calling a function on a DataFrame column with inplace=True may or may not work. This is especially true when chained indexing is involved. As if the problems described above aren't enough, inplace=True also hinders method chaining. Contrast the working of
```
result = df.some_function1().reset_index().some_function2()
```
As opposed to
```
temp = df.some_function1()
temp.reset_index(inplace=True)
result = temp.some_function2()
```
The former lends itself to better code organization and readability. Another supporting claim is that the API for set_axis was recently changed such that inplace default value was switched from True to False. See GH27600. Great job devs!


Question:
How to join two dataframes for which column values are within a certain range? Given two dataframes df_1 and df_2, how to join them such that datetime column df_1 is in between start and end in dataframe df_2:
```
print df_1

  timestamp              A          B
0 2016-05-14 10:54:33    0.020228   0.026572
1 2016-05-14 10:54:34    0.057780   0.175499
2 2016-05-14 10:54:35    0.098808   0.620986
3 2016-05-14 10:54:36    0.158789   1.014819
4 2016-05-14 10:54:39    0.038129   2.384590

print df_2

  start                end                  event
0 2016-05-14 10:54:31  2016-05-14 10:54:33  E1
1 2016-05-14 10:54:34  2016-05-14 10:54:37  E2
2 2016-05-14 10:54:38  2016-05-14 10:54:42  E3
```
Get corresponding event where df1.timestamp is between df_2.start and df2.end
```
timestamp              A          B          event
0 2016-05-14 10:54:33    0.020228   0.026572   E1
1 2016-05-14 10:54:34    0.057780   0.175499   E2
2 2016-05-14 10:54:35    0.098808   0.620986   E2
3 2016-05-14 10:54:36    0.158789   1.014819   E2
4 2016-05-14 10:54:39    0.038129   2.384590   E3
```

Response:
One simple solution is create interval index from start and end setting closed = both then use get_loc to get the event i.e (Hope all the date times are in timestamps dtype )
```
df_2.index = pd.IntervalIndex.from_arrays(df_2['start'],df_2['end'],closed='both')
df_1['event'] = df_1['timestamp'].apply(lambda x : df_2.iloc[df_2.index.get_loc(x)]['event'])
```
Output :
```
timestamp         A         B event
0 2016-05-14 10:54:33  0.020228  0.026572    E1
1 2016-05-14 10:54:34  0.057780  0.175499    E2
2 2016-05-14 10:54:35  0.098808  0.620986    E2
3 2016-05-14 10:54:36  0.158789  1.014819    E2
4 2016-05-14 10:54:39  0.038129  2.384590    E3
```


Question:
How to remove rows in a Pandas dataframe if the same row exists in another dataframe? I have two dataframes:
```
df1 = row1;row2;row3
 df2 = row4;row5;row6;row2
```
I want my output dataframe to only contain the rows unique in df1, i.e.:
```
df_out = row1;row3
```
How do I get this most efficiently? This code does what I want, but using 2 for-loops:
```
a = pd.DataFrame({0:[1,2,3],1:[10,20,30]})
b = pd.DataFrame({0:[0,1,2,3],1:[0,1,20,3]})

match_ident = []
for i in range(0,len(a)):
    found=False
    for j in range(0,len(b)):
        if a[0][i]==b[0][j]:
            if a[1][i]==b[1][j]:
                found=True
    match_ident.append(not(found))

a = a[match_ident]
```

Response:
You an use merge with parameter indicator and outer join, query for filtering and then remove helper column with drop: DataFrames are joined on all columns, so on parameter can be omit.
```
print (pd.merge(a,b, indicator=True, how='outer')
         .query('_merge=="left_only"')
         .drop('_merge', axis=1))
   0   1
0  1  10
2  3  30
```


Question:
How to drop a row whose particular column is empty/NaN? I have a csv file. I read it:
```
import pandas as pd
data = pd.read_csv('my_data.csv', sep=',')
data.head()
```
It has output like:
```
id    city    department    sms    category
01    khi      revenue      NaN       0
02    lhr      revenue      good      1
03    lhr      revenue      NaN       0
```
I want to remove all the rows where sms column is empty/NaN. What is efficient way to do it?

Response:
Use dropna with parameter subset for specify column for check NaNs:
```
data = data.dropna(subset=['sms'])
print (data)
   id city department   sms  category
1   2  lhr    revenue  good         1
```
Another solution with boolean indexing and notnull:
```
data = data[data['sms'].notnull()]
print (data)
   id city department   sms  category
1   2  lhr    revenue  good         1
```
Alternative with query:
```
print (data.query("sms == sms"))
   id city department   sms  category
1   2  lhr    revenue  good         1
```
Timings
```
#[300000 rows x 5 columns]
data = pd.concat([data]*100000).reset_index(drop=True)

```python
%timeit (data.dropna(subset=['sms']))
100 loops, best of 3: 19.5 ms per loop
```

```python
%timeit (data[data['sms'].notnull()])
100 loops, best of 3: 13.8 ms per loop
```

```python
%timeit (data.query("sms == sms"))
10 loops, best of 3: 23.6 ms per loop
```
```


Question:
Selecting columns by list (and columns are subset of list) I'm selecting several columns of a dataframe, by a list of the column names. This works fine if all elements of the list are in the dataframe. But if some elements of the list are not in the DataFrame, then it will generate the error "not in index". Is there a way to select all columns which included in that list, even if not all elements of the list are included in the dataframe? Here is some sample data which generates the above error:
```
df   = pd.DataFrame( [[0,1,2]], columns=list('ABC') )

lst  = list('ARB')

data = df[lst]       # error: not in index
```

Response:
I think you need Index.intersection:
```
df = pd.DataFrame({'A':[1,2,3],
                   'B':[4,5,6],
                   'C':[7,8,9],
                   'D':[1,3,5],
                   'E':[5,3,6],
                   'F':[7,4,3]})

print (df)
   A  B  C  D  E  F
0  1  4  7  1  5  7
1  2  5  8  3  3  4
2  3  6  9  5  6  3

lst = ['A','R','B']

print (df.columns.intersection(lst))
Index(['A', 'B'], dtype='object')

data = df[df.columns.intersection(lst)]
print (data)
   A  B
0  1  4
1  2  5
2  3  6
```
Another solution with numpy.intersect1d:
```
data = df[np.intersect1d(df.columns, lst)]
print (data)
   A  B
0  1  4
1  2  5
2  3  6
```


Question:
How to drop a list of rows from Pandas dataframe? I have a dataframe df :
```

```python
df
#Output
#                  sales  discount  net_sales    cogs
#STK_ID RPT_Date
#600141 20060331   2.709       NaN      2.709   2.245
#       20060630   6.590       NaN      6.590   5.291
#       20060930  10.103       NaN     10.103   7.981
#       20061231  15.915       NaN     15.915  12.686
#       20070331   3.196       NaN      3.196   2.710
#       20070630   7.907       NaN      7.907   6.459
#```
#Then I want to drop rows with certain sequence numbers which indicated in a list, suppose here is [1,2,4], then left:
#```
#sales  discount  net_sales    cogs
#STK_ID RPT_Date
#600141 20060331   2.709       NaN      2.709   2.245
#       20061231  15.915       NaN     15.915  12.686
#       20070630   7.907       NaN      7.907   6.459
#```
#How or what function can do that ?
```

Response:
Use DataFrame.drop and pass it a Series of index labels:
```

```python
df
#Output
#one  two
#one      1    4
#two      2    3
#three    3    2
#four     4    1
```

```python
df.drop(df.index[[1,3]])
#Output
#one  two
#one      1    4
#three    3    2
#```
```


Question:
Filtering all rows with NaT in a column in Dataframe python I have a df like this:
```
a b           c
    1 NaT         w
    2 2014-02-01  g
    3 NaT         x

    df=df[df.b=='2014-02-01']
```
will give me
```
a  b          c
    2 2014-02-01  g
```
I want a database of all rows with NaT in column b?
```
df=df[df.b==None] #Doesn't work
```
I want this:
```
a b           c
    1 NaT         w
    3 NaT         x
```

Response:
isnull and notnull work with NaT so you can handle them much the same way you handle NaNs:
```

```python
df
```

a          b  c
0  1        NaT  w
1  2 2014-02-01  g
2  3        NaT  x

```python
df.dtypes
```

a             int64
b    datetime64[ns]
c            object
```
just use isnull to select:
```
df[df.b.isnull()]

   a   b  c
0  1 NaT  w
2  3 NaT  x
```


Question:
Efficient way to apply multiple filters to pandas DataFrame or Series I have a scenario where a user wants to apply several filters to a Pandas DataFrame or Series object. Essentially, I want to efficiently chain a bunch of filtering (comparison operations) together that are specified at run-time by the user. The filters should be additive (aka each one applied should narrow results). I'm currently using reindex() (as below) but this creates a new object each time and copies the underlying data (if I understand the documentation correctly). I want to avoid this unnecessary copying as it will be really inefficient when filtering a big Series or DataFrame. I'm thinking that using apply(), map(), or something similar might be better. I'm pretty new to Pandas though so still trying to wrap my head around everything. Also, I would like to expand this so that the dictionary passed in can include the columns to operate on and filter an entire DataFrame based on the input dictionary. However, I'm assuming whatever works for a Series can be easily expanded to a DataFrame. TL;DR I want to take a dictionary of the following form and apply each operation to a given Series object and return a 'filtered' Series object.
```
relops = {'>=': [1], '<=': [1]}
```
Long Example I'll start with an example of what I have currently and just filtering a single Series object. Below is the function I'm currently using:
```
def apply_relops(series, relops):
        """
        Pass dictionary of relational operators to perform on given series object
        """
        for op, vals in relops.iteritems():
            op_func = ops[op]
            for val in vals:
                filtered = op_func(series, val)
                series = series.reindex(series[filtered])
        return series
```
The user provides a dictionary with the operations they want to perform:
```

```python
df = pandas.DataFrame({'col1': [0, 1, 2], 'col2': [10, 11, 12]})
```

```python
print df
```

```python
print df
#Output
#   col1  col2
#0     0    10
#1     1    11
#2     2    12
```

```python
from operator import le, ge
```

```python
ops ={'>=': ge, '<=': le}
```

```python
apply_relops(df['col1'], {'>=': [1]})
#Output
#col1
#1       1
#2       2
#Name: col1
```

```python
apply_relops(df['col1'], relops = {'>=': [1], '<=': [1]})
#Output
#col1
#1       1
#Name: col1
#```
#Again, the 'problem' with my above approach is that I think there is a lot of possibly unnecessary copying of the data for the in-between steps.
```

Response:
Pandas (and numpy) allow for boolean indexing, which will be much more efficient:
```

```python
df.loc[df['col1'] >= 1, 'col1']
#Output
#1    1
#2    2
#Name: col1
```

```python
df[df['col1'] >= 1]
#Output
#col1  col2
#1     1    11
#2     2    12
```

```python
df[(df['col1'] >= 1) & (df['col1'] <=1 )]
#Output
#col1  col2
#1     1    11
#```
#If you want to write helper functions for this, consider something along these lines:
#```
```

```python
def b(x, col, op, n):
             return op(x[col],n)
```

```python
def f(x, *b):
             return x[(np.logical_and(*b))]
```

```python
b1 = b(df, 'col1', ge, 1)
```

```python
b2 = b(df, 'col1', le, 1)
```

```python
f(df, b1, b2)
#Output
#col1  col2
#1     1    11
#```
#Update: pandas 0.13 has a query method for these kind of use cases, assuming column names are valid identifiers the following works (and can be more efficient for large frames as it uses numexpr behind the scenes):
#```
```

```python
df.query('col1 <= 1 & 1 <= col1')
#Output
#col1  col2
#1     1    11
#```
```