Question:
pandas how to swap or reorder columns I know that there are ways to swap the column order in python pandas. Let say I have this example dataset:
```
import pandas as pd
employee = {'EmployeeID' : [0,1,2],
     'FirstName' : ['a','b','c'],
     'LastName' : ['a','b','c'],
     'MiddleName' : ['a','b', None],
     'Contact' : ['(M) 133-245-3123', '(F)
[email protected]
', '(F)312-533-2442
[email protected]
']}

df = pd.DataFrame(employee)
```
The one basic way to do would be:
```
neworder = ['EmployeeID','FirstName','MiddleName','LastName','Contact']
df=df.reindex(columns=neworder)
```
However, as you can see, I only want to swap two columns. It was doable just because there are only 4 column, but what if I have like 100 columns? what would be an effective way to swap or reorder columns? There might be 2 cases: when you just want 2 columns swapped. when you want 3 columns reordered. (I am pretty sure that this case can be applied to more than 3 columns.)

Response:
Say your current order of column is [b,c,d,a] and you want to order it into [a,b,c,d], you could do it this way:
```
new_df = old_df[['a', 'b', 'c', 'd']]
```


Question: Pandas: unique dataframe I have a DataFrame that has duplicated rows. I'd like to get a DataFrame with a unique index and no duplicates. It's ok to discard the duplicated values. Is this possible? Would it be a done by groupby?

Response:
```

```python
df.drop_duplicates()
#Output
#b  c
#1  2  3
#3  4  0
#7  5  9
#```
```


Question:
Trouble passing in lambda to apply for pandas DataFrame: "TypeError: () got an unexpected keyword argument 'axis' " I'm trying to apply a function to all rows of a pandas DataFrame (actually just one column in that DataFrame) I'm sure this is a syntax error but I'm know sure what I'm doing wrong
```py
df['col'].apply(lambda x, y:(x - y).total_seconds(), args=[d1], axis=1)
```
The col column contains a bunch a datetime.datetime objects and d1 is the earliest of them. I'm trying to get a column of the total number of seconds for each of the rows. I keep getting the following error
```none
TypeError: <lambda>() got an unexpected keyword argument 'axis'
```
I don't understand why axis is getting passed to my lambda function I've also tried doing
```py
def diff_dates(d1, d2):
    return (d1-d2).total_seconds()

df['col'].apply(diff_dates, args=[d1], axis=1)
```
And I get the same error.

Response: Note there is no axis param for a Series.apply call, as distinct to a DataFrame.apply call. Series.apply(func, convert_dtype=True, args=(), **kwds) ... func : function convert_dtype : boolean, default True Try to find better dtype for elementwise function results. If False, leave as dtype=object args : tuple Positional arguments to pass to function in addition to the value **kwds Additional keyword arguments passed to func. There is one for a df but it's unclear how you're expecting this to work when you're calling it on a series but you're expecting it to work on a row?


Question:
Pandas monthly rolling operation I ended up figuring it out while writing out this question so I'll just post anyway and answer my own question in case someone else needs a little help. Problem Suppose we have a DataFrame, df, containing this data.
```
import pandas as pd
from io import StringIO

data = StringIO(
"""\
date          spendings  category
2014-03-25    10         A
2014-04-05    20         A
2014-04-15    10         A
2014-04-25    10         B
2014-05-05    10         B
2014-05-15    10         A
2014-05-25    10         A
"""
)

df = pd.read_csv(data,sep="\s+",parse_dates=True,index_col="date")
```
Goal For each row, sum the spendings over every row that is within one month of it, ideally using DataFrame.rolling as it's a very clean syntax. What I have tried
```
df = df.rolling("M").sum()
```
But this throws an exception
```
ValueError: <MonthEnd> is a non-fixed frequency
```
version: pandas==0.19.2

Response:
Use the "D" offset rather than "M" and specifically use "30D" for 30 days or approximately one month.
```
df = df.rolling("30D").sum()
```
Initially, I intuitively jumped to using "M" as I figured it stands for one month, but now it's clear why that doesn't work.


Question:
Pandas: add a column to a multiindex column dataframe I would like to add a column to the second level of a multiindex column dataframe.
```

```python
df
#Output
#first        bar                 baz
#second       one       two       one       two
#A       0.487880 -0.487661 -1.030176  0.100813
#B       0.267913  1.918923  0.132791  0.178503
#C       1.550526 -0.312235 -1.177689 -0.081596
#```
#The usual trick of direct assignment does not work:
#```
```

```python
df['bar']['three'] = [0, 1, 2]
```

```python
df
#Output
#first        bar                 baz
#second       one       two       one       two
#A       0.487880 -0.487661 -1.030176  0.100813
#B       0.267913  1.918923  0.132791  0.178503
#C       1.550526 -0.312235 -1.177689 -0.081596
#```
#How can I add the third row to under "bar"?
```

Response:
It's actually pretty simple (FWIW, I originally thought to do it your way):
```
df['bar', 'three'] = [0, 1, 2]
df = df.sort_index(axis=1)
print(df)

        bar                        baz
        one       two  three       one       two
A -0.212901  0.503615      0 -1.660945  0.446778
B -0.803926 -0.417570      1 -0.336827  0.989343
C  3.400885 -0.214245      2  0.895745  1.011671
```


Question:
How can I read tar.gz file using pandas read_csv with gzip compression option? I have a very simple csv, with the following data, compressed inside the tar.gz file. I need to read that in dataframe using pandas.read_csv.
```
A  B
0  1  4
1  2  5
2  3  6

import pandas as pd
pd.read_csv("sample.tar.gz",compression='gzip')
```
However, I am getting error:
```
CParserError: Error tokenizing data. C error: Expected 1 fields in line 440, saw 2
```
Following are the set of read_csv commands and the different errors I get with them:
```
pd.read_csv("sample.tar.gz",compression='gzip',  engine='python')
Error: line contains NULL byte

pd.read_csv("sample.tar.gz",compression='gzip', header=0)
CParserError: Error tokenizing data. C error: Expected 1 fields in line 440, saw 2

pd.read_csv("sample.tar.gz",compression='gzip', header=0, sep=" ")
CParserError: Error tokenizing data. C error: Expected 2 fields in line 94, saw 14

pd.read_csv("sample.tar.gz",compression='gzip', header=0, sep=" ", engine='python')
Error: line contains NULL byte
```
What's going wrong here? How can I fix this?

Response:
```
df = pd.read_csv('sample.tar.gz', compression='gzip', header=0, sep=' ', quotechar='"', error_bad_lines=False)
```
Note: error_bad_lines=False will ignore the offending rows.


Question:
How do I get the name of the rows from the index of a data frame? Consider a data frame with row names that aren't a column of their own per se, such as the following:
```
X  Y
 Row 1  0  5
 Row 2  8  1
 Row 3  3  0
```
How would I extract the name of these rows as a list, if I have their index? For example, it would look something like:
```
function_name(dataframe[indices])
> ['Row 1', 'Row 2']
```

Response: df.index outputs the row names as pandas Index object. list(df.index) casts to a list. df.index['Row 2':'Row 5'] supports label slicing similar to columns.


Question:
AttributeError: 'Series' object has no attribute 'reshape' I'm using sci-kit learn linear regression algorithm. While scaling Y target feature with:
```
Ys = scaler.fit_transform(Y)
```
I got ValueError: Expected 2D array, got 1D array instead: After that I reshaped using:
```
Ys = scaler.fit_transform(Y.reshape(-1,1))
```
But got error again: AttributeError: 'Series' object has no attribute 'reshape' So I checked pandas.Series documentation page and it says: reshape(*args, **kwargs) Deprecated since version 0.19.0.

Response:
Solution was linked on reshaped method on documentation page. Insted of Y.reshape(-1,1) you need to use:
```
Y.values.reshape(-1,1)
```


Question:
Find all columns of dataframe in Pandas whose type is float, or a particular type? I have a dataframe, df, that has some columns of type float64, while the others are of object. Due to the mixed nature, I cannot use
```
df.fillna('unknown') #getting error "ValueError: could not convert string to float:"
```
as the error happened with the columns whose type is float64 (what a misleading error message!) so I'd wish that I could do something like
```
for col in df.columns[<dtype == object>]:
    df[col] = df[col].fillna("unknown")
```
So my question is if there is any such filter expression that I can use with df.columns? I guess alternatively, less elegantly, I could do:
```
for col in df.columns:
        if (df[col].dtype == dtype('O')): # for object type
            df[col] = df[col].fillna('')
            # still puzzled, only empty string works as replacement, 'unknown' would not work for certain value leading to error of "ValueError: Error parsing datetime string "unknown" at position 0"
```
I also would like to know why in the above code replacing '' with 'unknown' the code would work for certain cells but failed with a cell with the error of "ValueError: Error parsing datetime string "unknown" at position 0" Thanks a lot! Yu

Response:
This is conciser:
```
# select the float columns
df_num = df.select_dtypes(include=[np.float])
# select non-numeric columns
df_num = df.select_dtypes(exclude=[np.number])
```


Question:
Search for a value anywhere in a pandas DataFrame This seems like a simple question, but I couldn't find it asked before (this and this are close but the answers aren't great). The question is: if I want to search for a value somewhere in my df (I don't know which column it's in) and return all rows with a match. What's the most Pandaic way to do it? Is there anything better than:
```
for col in list(df):
    try:
        df[col] == var
        return df[df[col] == var]
    except TypeError:
        continue
```
?

Response:
You can perform equality comparison on the entire DataFrame:
```
df[df.eq(var1).any(1)]
```


Question:
How to combine multiple rows of strings into one using pandas? I have a DataFrame with multiple rows. Is there any way in which they can be combined to form one string? For example:
```
words
0    I, will, hereby
1    am, gonna
2    going, far
3    to
4    do
5    this
```
Expected output:
```
I, will, hereby, am, gonna, going, far, to, do, this
```

Response:
You can use str.cat to join the strings in each row. For a Series or column s, write:
```

```python
s.str.cat(sep=', ')
#Output
#'I, will, hereby, am, gonna, going, far, to, do, this'
#```
```


Question:
Indexing Pandas data frames: integer rows, named columns Say df is a pandas dataframe. df.loc[] only accepts names df.iloc[] only accepts integers (actual placements) df.ix[] accepts both names and integers: When referencing rows, df.ix[row_idx, ] only wants to be given names. e.g.
```
df = pd.DataFrame({'a' : ['one', 'two', 'three','four', 'five', 'six'],
                   '1' : np.arange(6)})
df = df.ix[2:6]
print(df)

   1      a
2  2  three
3  3   four
4  4   five
5  5    six

df.ix[0, 'a']
```
throws an error, it doesn't give return 'two'. When referencing columns, iloc is prefers integers, not names. e.g.
```
df.ix[2, 1]
```
returns 'three', not 2. (Although df.idx[2, '1'] does return 2). Oddly, I'd like the exact opposite functionality. Usually my column names are very meaningful, so in my code I reference them directly. But due to a lot of observation cleaning, the row names in my pandas data frames don't usually correspond to range(len(df)). I realize I can use:
```
df.iloc[0].loc['a'] # returns three
```
But it seems ugly! Does anyone know of a better way to do this, so that the code would look like this?
```
df.foo[0, 'a'] # returns three
```
In fact, is it possible to add on my own new method to pandas.core.frame.DataFrames, so e.g. df.idx(rows, cols) is in fact df.iloc[rows].loc[cols]?

Response: It's a late answer, but @unutbu's comment is still valid and a great solution to this problem. To index a DataFrame with integer rows and named columns (labeled columns): df.loc[df.index[#], 'NAME'] where # is a valid integer index and NAME is the name of the column.


Question:
Concat DataFrame Reindexing only valid with uniquely valued Index objects I am trying to concat the following dataframes: df1
```
price   side timestamp
timestamp
2016-01-04 00:01:15.631331072   0.7286  2   1451865675631331
2016-01-04 00:01:15.631399936   0.7286  2   1451865675631400
2016-01-04 00:01:15.631860992   0.7286  2   1451865675631861
2016-01-04 00:01:15.631866112   0.7286  2   1451865675631866
```
and: df2
```
bid     bid_size offer  offer_size
timestamp
2016-01-04 00:00:31.331441920   0.7284  4000000 0.7285  1000000
2016-01-04 00:00:53.631324928   0.7284  4000000 0.7290  4000000
2016-01-04 00:01:03.131234048   0.7284  5000000 0.7286  4000000
2016-01-04 00:01:12.131444992   0.7285  1000000 0.7286  4000000
2016-01-04 00:01:15.631364096   0.7285  4000000 0.7290  4000000
```
With
```
data = pd.concat([df1,df2], axis=1)
```
But I get the follwing output:
```
InvalidIndexError                         Traceback (most recent call last)
<ipython-input-38-2e88458f01d7> in <module>()
----> 1 data = pd.concat([df1,df2], axis=1)
      2 data = data.fillna(method='pad')
      3 data = data.fillna(method='bfill')
      4 data['timestamp'] =  data.index.values#converting to datetime
      5 data['timestamp'] = pd.to_datetime(data['timestamp'])#converting to datetime

/usr/local/lib/python2.7/site-packages/pandas/tools/merge.pyc in concat(objs, axis, join, join_axes, ignore_index, keys, levels, names, verify_integrity, copy)
    810                        keys=keys, levels=levels, names=names,
    811                        verify_integrity=verify_integrity,
--> 812                        copy=copy)
    813     return op.get_result()
    814

/usr/local/lib/python2.7/site-packages/pandas/tools/merge.pyc in __init__(self, objs, axis, join, join_axes, keys, levels, names, ignore_index, verify_integrity, copy)
    947         self.copy = copy
    948
--> 949         self.new_axes = self._get_new_axes()
    950
    951     def get_result(self):

/usr/local/lib/python2.7/site-packages/pandas/tools/merge.pyc in _get_new_axes(self)
   1013                 if i == self.axis:
   1014                     continue
-> 1015                 new_axes[i] = self._get_comb_axis(i)
   1016         else:
   1017             if len(self.join_axes) != ndim - 1:

/usr/local/lib/python2.7/site-packages/pandas/tools/merge.pyc in _get_comb_axis(self, i)
   1039                 raise TypeError("Cannot concatenate list of %s" % types)
   1040
-> 1041         return _get_combined_index(all_indexes, intersect=self.intersect)
   1042
   1043     def _get_concat_axis(self):

/usr/local/lib/python2.7/site-packages/pandas/core/index.pyc in _get_combined_index(indexes, intersect)
   6120             index = index.intersection(other)
   6121         return index
-> 6122     union = _union_indexes(indexes)
   6123     return _ensure_index(union)
   6124

/usr/local/lib/python2.7/site-packages/pandas/core/index.pyc in _union_indexes(indexes)
   6149
   6150         if hasattr(result, 'union_many'):
-> 6151             return result.union_many(indexes[1:])
   6152         else:
   6153             for other in indexes[1:]:

/usr/local/lib/python2.7/site-packages/pandas/tseries/index.pyc in union_many(self, others)
    959             else:
    960                 tz = this.tz
--> 961                 this = Index.union(this, other)
    962                 if isinstance(this, DatetimeIndex):
    963                     this.tz = tz

/usr/local/lib/python2.7/site-packages/pandas/core/index.pyc in union(self, other)
   1553                 result.extend([x for x in other._values if x not in value_set])
   1554         else:
-> 1555             indexer = self.get_indexer(other)
   1556             indexer, = (indexer == -1).nonzero()
   1557

/usr/local/lib/python2.7/site-packages/pandas/core/index.pyc in get_indexer(self, target, method, limit, tolerance)
   1890
   1891         if not self.is_unique:
-> 1892             raise InvalidIndexError('Reindexing only valid with uniquely'
   1893                                     ' valued Index objects')
   1894

InvalidIndexError: Reindexing only valid with uniquely valued Index objects
```
I have removed additional columns and removed duplicates and NA where there could be a conflict - but I simply do not know what's wrong.

Response: Duplicated column names! In my case the problem was because I had duplicated column names.


Question:
Drop all duplicate rows across multiple columns in Python Pandas The pandas drop_duplicates function is great for "uniquifying" a dataframe. I would like to drop all rows which are duplicates across a subset of columns. Is this possible?
```none
A   B   C
0   foo 0   A
1   foo 1   A
2   foo 1   B
3   bar 1   A
```
As an example, I would like to drop rows which match on columns A and C so this should drop rows 0 and 1.

Response:
This is much easier in pandas now with drop_duplicates and the keep parameter.
```
import pandas as pd
df = pd.DataFrame({"A":["foo", "foo", "foo", "bar"], "B":[0,1,1,1], "C":["A","A","B","A"]})
df.drop_duplicates(subset=['A', 'C'], keep=False)
```


Question: Find the max of two or more columns with pandas I have a dataframe with columns A,B. I need to create a column C such that for every record / row: C = max(A, B). How should I go about doing this?

Response:
You can get the maximum like this:
```

```python
import pandas as pd
```

```python
df = pd.DataFrame({"A": [1,2,3], "B": [-2, 8, 1]})
```

```python
df
#Output
#   A  B
#0  1 -2
#1  2  8
#2  3  1
```

```python
df[["A", "B"]]
#Output
#   A  B
#0  1 -2
#1  2  8
#2  3  1
```

```python
df[["A", "B"]].max(axis=1)
#Output
#0    1
#1    8
#2    3
#```
#and so:
#```
```

```python
df["C"] = df[["A", "B"]].max(axis=1)
```

```python
df
#Output
#   A  B  C
#0  1 -2  1
#1  2  8  8
#2  3  1  3
#```
#If you know that "A" and "B" are the only columns, you could even get away with
#```
```

```python
df["C"] = df.max(axis=1)
#Output
#```
#And you could use .apply(max, axis=1) too, I guess.
```


Question:
Python Pandas GroupBy get list of groups I have a line of code:
```
g = x.groupby('Color')
```
The colors are Red, Blue, Green, Yellow, Purple, Orange, and Black. How do I return this list? For similar attributes, I use x.Attribute and it works fine, but x.Color doesn't behave the same way.

Response:
There is much easier way of doing it:
```
g = x.groupby('Color')

g.groups.keys()
```
By doing groupby() pandas returns you a dict of grouped DFs. You can easily get the key list of this dict by python built in function keys().


Question: Delete the first three rows of a dataframe in pandas I need to delete the first three rows of a dataframe in pandas. I know df.ix[:-1] would remove the last row, but I can't figure out how to remove first n rows.

Response:
Use iloc:
```
df = df.iloc[3:]
```
will give you a new df without the first three rows.


Question:
How to reset index in a pandas dataframe? I have a dataframe from which I remove some rows. As a result, I get a dataframe in which index is something like [1,5,6,10,11] and I would like to reset it to [0,1,2,3,4]. How can I do it? The following seems to work:
```py
df = df.reset_index()
del df['index']
```
The following does not work:
```py
df = df.reindex()
```

Response:
DataFrame.reset_index is what you're looking for. If you don't want it saved as a column, then do:
```
df = df.reset_index(drop=True)
```
If you don't want to reassign:
```
df.reset_index(drop=True, inplace=True)
```


Question:
How to check if a column exists in Pandas How do I check if a column exists in a Pandas DataFrame df?
```none
A   B    C
0  3  40  100
1  6  30  200
```
How would I check if the column "A" exists in the above DataFrame so that I can compute:
```py
df['sum'] = df['A'] + df['C']
```
And if "A" doesn't exist:
```py
df['sum'] = df['B'] + df['C']
```

Response:
This will work:
```
if 'A' in df:
```
But for clarity, I'd probably write it as:
```
if 'A' in df.columns:
```


Question:
Adding a column thats result of difference in consecutive rows in pandas Lets say I have a dataframe like this
```
A   B
0   a   b
1   c   d
2   e   f
3   g   h
```
0,1,2,3 are times, a, c, e, g is one time series and b, d, f, h is another time series. I need to be able to add two columns to the orignal dataframe which is got by computing the differences of consecutive rows for certain columns. So i need something like this
```
A   B   dA
0   a   b  (a-c)
1   c   d  (c-e)
2   e   f  (e-g)
3   g   h   Nan
```
I saw something called diff on the dataframe/series but that does it slightly differently as in first element will become Nan.

Response:
Use shift.
```
df['dA'] = df['A'] - df['A'].shift(-1)
```


Question:
Get a list from Pandas DataFrame column headers I want to get a list of the column headers from a Pandas DataFrame. The DataFrame will come from user input, so I won't know how many columns there will be or what they will be called. For example, if I'm given a DataFrame like this:
```none
y  gdp  cap
0   1    2    5
1   2    3    9
2   8    7    2
3   3    4    7
4   6    7    7
5   4    8    3
6   8    2    8
7   9    9   10
8   6    6    4
9  10   10    7
```
I would get a list like this:
```none
['y', 'gdp', 'cap']
```

Response:
You can get the values as a list by doing:
```
list(my_dataframe.columns.values)
```
Also you can simply use (as shown in Ed Chum's answer):
```
list(my_dataframe)
```


Question:
Find element's index in pandas Series I know this is a very basic question but for some reason I can't find an answer. How can I get the index of certain element of a Series in python pandas? (first occurrence would suffice) I.e., I'd like something like:
```
import pandas as pd
myseries = pd.Series([1,4,0,7,5], index=[0,1,2,3,4])
print myseries.find(7) # should output 3
```
Certainly, it is possible to define such a method with a loop:
```
def find(s, el):
    for i in s.index:
        if s[i] == el:
            return i
    return None

print find(myseries, 7)
```
but I assume there should be a better way. Is there?

Response:
```

```python
myseries[myseries == 7]
#Output
#3    7
#dtype: int64
```

```python
myseries[myseries == 7].index[0]
#Output
#3
#```
#Though I admit that there should be a better way to do that, but this at least avoids iterating and looping through the object and moves it to the C level.
```


Question:
pandas: merge (join) two data frames on multiple columns I am trying to join two pandas dataframes using two columns:
```py
new_df = pd.merge(A_df, B_df,  how='left', left_on='[A_c1,c2]', right_on = '[B_c1,c2]')
```
but got the following error:
```none
pandas/index.pyx in pandas.index.IndexEngine.get_loc (pandas/index.c:4164)()

pandas/index.pyx in pandas.index.IndexEngine.get_loc (pandas/index.c:4028)()

pandas/src/hashtable_class_helper.pxi in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13166)()

pandas/src/hashtable_class_helper.pxi in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13120)()

KeyError: '[B_1, c2]'
```
Any idea what should be the right way to do this?

Response:
Try this
```
new_df = pd.merge(
    left=A_df,
    right=B_df,
    how='left',
    left_on=['A_c1', 'c2'],
    right_on=['B_c1', 'c2'],
)
```
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.merge.html left_on : label or list, or array-like Field names to join on in left DataFrame. Can be a vector or list of vectors of the length of the DataFrame to use a particular vector as the join key instead of columns right_on : label or list, or array-like Field names to join on in right DataFrame or vector/list of vectors per left_on docs


Question:
Insert a Pandas Dataframe into mongodb using PyMongo What is the quickest way to insert a pandas DataFrame into mongodb using PyMongo? Attempts
```
db.myCollection.insert(df.to_dict())
```
gave an error InvalidDocument: documents must have only string keys, the key was Timestamp('2013-11-23 13:31:00', tz=None)
```
db.myCollection.insert(df.to_json())
```
gave an error TypeError: 'str' object does not support item assignment
```
db.myCollection.insert({id: df.to_json()})
```
gave an error InvalidDocument: documents must have only string a keys, key was  df
```
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 150 entries, 2013-11-23 13:31:26 to 2013-11-23 13:24:07
Data columns (total 3 columns):
amount    150  non-null values
price     150  non-null values
tid       150  non-null values
dtypes: float64(2), int64(1)
```

Response:
Here you have the very quickest way. Using the insert_many method from pymongo 3 and 'records' parameter of to_dict method.
```
db.collection.insert_many(df.to_dict('records'))
```


Question:
How to drop rows of Pandas DataFrame whose value in a certain column is NaN I have this DataFrame and want only the records whose EPS column is not NaN:
```none
STK_ID  EPS  cash
STK_ID RPT_Date
601166 20111231  601166  NaN   NaN
600036 20111231  600036  NaN    12
600016 20111231  600016  4.3   NaN
601009 20111231  601009  NaN   NaN
601939 20111231  601939  2.5   NaN
000001 20111231  000001  NaN   NaN
```
...i.e. something like df.drop(....) to get this resulting dataframe:
```none
STK_ID  EPS  cash
STK_ID RPT_Date
600016 20111231  600016  4.3   NaN
601939 20111231  601939  2.5   NaN
```
How do I do that?

Response:
This question is already resolved, but... ...also consider the solution suggested by Wouter in his original comment. The ability to handle missing data, including dropna(), is built into pandas explicitly. Aside from potentially improved performance over doing it manually, these functions also come with a variety of options which may be useful.
```

```python
df = pd.DataFrame(np.random.randn(10,3))
```

```python
df.iloc[::2,0] = np.nan; df.iloc[::4,1] = np.nan; df.iloc[::3,2] = np.nan;
```

```python
df
#Output
#0         1         2
#0       NaN       NaN       NaN
#1  2.677677 -1.466923 -0.750366
#2       NaN  0.798002 -0.906038
#3  0.672201  0.964789       NaN
#4       NaN       NaN  0.050742
#5 -1.250970  0.030561 -2.678622
#6       NaN  1.036043       NaN
#7  0.049896 -0.308003  0.823295
#8       NaN       NaN  0.637482
#9 -0.310130  0.078891       NaN
#```
#```
```

```python
df.dropna()     #drop all rows that have any NaN values
#Output
#0         1         2
#1  2.677677 -1.466923 -0.750366
#5 -1.250970  0.030561 -2.678622
#7  0.049896 -0.308003  0.823295
#```
#```
```

```python
df.dropna(how='all')     #drop only if ALL columns are NaN
#Output
#0         1         2
#1  2.677677 -1.466923 -0.750366
#2       NaN  0.798002 -0.906038
#3  0.672201  0.964789       NaN
#4       NaN       NaN  0.050742
#5 -1.250970  0.030561 -2.678622
#6       NaN  1.036043       NaN
#7  0.049896 -0.308003  0.823295
#8       NaN       NaN  0.637482
#9 -0.310130  0.078891       NaN
#```
#```
```

```python
df.dropna(thresh=2)   #Drop row if it does not have at least two values that are **not** NaN
#Output
#0         1         2
#1  2.677677 -1.466923 -0.750366
#2       NaN  0.798002 -0.906038
#3  0.672201  0.964789       NaN
#5 -1.250970  0.030561 -2.678622
#7  0.049896 -0.308003  0.823295
#9 -0.310130  0.078891       NaN
#```
#```
```

```python
df.dropna(subset=[1])   #Drop only if NaN in specific column (as asked in the question)
#Output
#0         1         2
#1  2.677677 -1.466923 -0.750366
#2       NaN  0.798002 -0.906038
#3  0.672201  0.964789       NaN
#5 -1.250970  0.030561 -2.678622
#6       NaN  1.036043       NaN
#7  0.049896 -0.308003  0.823295
#9 -0.310130  0.078891       NaN
#```
#There are also other options (See docs at http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.dropna.html), including dropping columns instead of rows. Pretty handy!
```


Question:
How to select rows that do not start with some str in pandas? I want to select rows that the values do not start with some str. For example, I have a pandas df, and I want to select data do not start with t, and c. In this sample, the output should be mext1 and okl1.
```
import pandas as pd

df=pd.DataFrame({'col':['text1','mext1','cext1','okl1']})
df

    col
0   text1
1   mext1
2   cext1
3   okl1
```
I want this:
```
col
0   mext1
1   okl1
```

Response:
You can use the str accessor to get string functionality. The get method can grab a given index of the string.
```
df[~df.col.str.get(0).isin(['t', 'c'])]

     col
1  mext1
3   okl1
```
Looks like you can use startswith as well with a tuple (and not a list) of the values you want to exclude.
```
df[~df.col.str.startswith(('t', 'c'))]
```


Question:
Pandas - combine column values into a list in a new column I have a Python Pandas dataframe df:
```
d = [['hello', 1, 'GOOD', 'long.kw'],
     [1.2, 'chipotle', np.nan, 'bingo'],
     ['various', np.nan, 3000, 123.456]]
t = pd.DataFrame(data=d, columns=['A','B','C','D'])
```
which looks like this:
```
print(t)
         A         B     C        D
0    hello         1  GOOD  long.kw
1      1.2  chipotle   NaN    bingo
2  various       NaN  3000  123.456
```
I am trying to create a new column which is a list of the values in A, B, C, and D. So it would look like this:
```
t['combined']

Out[125]:
0        [hello, 1, GOOD, long.kw]
1        [1.2, chipotle, nan, bingo]
2        [various, nan, 3000, 123.456]
Name: combined, dtype: object
```
I am trying this code:
```
t['combined'] = t.apply(lambda x: list([x['A'],
                                        x['B'],
                                        x['C'],
                                        x['D']]),axis=1)
```
Which returns this error:
```
ValueError: Wrong number of items passed 4, placement implies 1
```
What is puzzling to me is if I remove one of the columns that I want to put in the list (or add another column to the dataframe that I DON'T add to the list), my code works. For instance, run this code:
```
t['combined'] = t.apply(lambda x: list([x['A'],
                                        x['B'],
                                        x['D']]),axis=1)
```
Returns this which is perfect if I only wanted the 3 columns:
```
print(t)
         A         B     C        D                 combined
0    hello         1  GOOD  long.kw      [hello, 1, long.kw]
1      1.2  chipotle   NaN    bingo   [1.2, chipotle, bingo]
2  various       NaN  3000  123.456  [various, nan, 123.456]
```
I am at a complete loss as to why requesting the 'combined' list be made of all columns in the dataframe would create an error, but selecting all but 1 column to create the 'combined' list works as expected.

Response:
try this :
```
t['combined']= t.values.tolist()

t
Out[50]:
         A         B     C        D                       combined
0    hello         1  GOOD  long.kw      [hello, 1, GOOD, long.kw]
1     1.20  chipotle   NaN    bingo    [1.2, chipotle, nan, bingo]
2  various       NaN  3000   123.46  [various, nan, 3000, 123.456]
```


Question:
Pandas: Get duplicated indexes Given a dataframe, I want to get the duplicated indexes, which do not have duplicate values in the columns, and see which values are different. Specifically, I have this dataframe:
```
import pandas as pd
wget https://www.dropbox.com/s/vmimze2g4lt4ud3/alt_exon_repeatmasker_intersect.bed
alt_exon_repeatmasker = pd.read_table('alt_exon_repeatmasker_intersect.bed', header=None, index_col=3)

```python
alt_exon_repeatmasker.index.is_unique
#Output
#False
#```
#And some of the indexes have duplicate values in the 9th column (the type of DNA repetitive element in this location), and I want to know what are the different types of repetitive elements for individual locations (each index = a genome location). I'm guessing this will require some kind of groupby and hopefully some groupby ninja can help me out. To simplify even further, if we only have the index and the repeat type,
#```
#genome_location1    MIR3
#genome_location1    AluJb
#genome_location2    Tigger1
#genome_location3    AT_rich
#```
#So the output I'd like to see all duplicate indexes and their repeat types, as such:
#```
#genome_location1    MIR3
#genome_location1    AluJb
#```
#EDIT: added toy example
```

Response: Also useful and very succinct: df[df.index.duplicated()] Note that this only returns one of the duplicated rows, so to see all the duplicated rows you'll want this: df[df.index.duplicated(keep=False)]


Question:
Combine two Pandas dataframes with the same index [duplicate] This question already has an answer here: What are the 'levels', 'keys', and names arguments for in Pandas' concat function? (1 answer) Closed 6 years ago. I have two dataframes with the same index but different columns. How do I combine them into one with the same index but containing all the columns? I have:
```
A
1 10
2 11

  B
1 20
2 21
```
and I need the following output:
```
A  B
1 10 20
2 11 21
```

Response:
```
pandas.concat([df1, df2], axis=1)
```


Question: Locate first and last non NaN values in a Pandas DataFrame I have a Pandas DataFrame indexed by date. There a number of columns but many columns are only populated for part of the time series. I'd like to find where the first and last values non-NaN values are located so that I can extracts the dates and see how long the time series is for a particular column. Could somebody point me in the right direction as to how I could go about doing something like this?

Response:
Here's some helpful examples. Series
```
s = pd.Series([np.NaN, 1, np.NaN, 3, np.NaN], index=list('abcde'))
s

a    NaN
b    1.0
c    NaN
d    3.0
e    NaN
dtype: float64

# first valid index
s.first_valid_index()
# 'b'

# first valid position
s.index.get_loc(s.first_valid_index())
# 1

# last valid index
s.last_valid_index()
# 'd'

# last valid position
s.index.get_loc(s.last_valid_index())
# 3
```
Alternative solution using notna and idxmax:
```
# first valid index
s.notna().idxmax()
# 'b'

# last valid index
s.notna()[::-1].idxmax()
# 'd'
```
DataFrame
```
df = pd.DataFrame({
    'A': [np.NaN, 1, np.NaN, 3, np.NaN],
    'B': [1, np.NaN, np.NaN, np.NaN, np.NaN]
})
df

     A    B
0  NaN  1.0
1  1.0  NaN
2  NaN  NaN
3  3.0  NaN
4  NaN  NaN
```
(first|last)_valid_index isn't defined on DataFrames, but you can apply them on each column using apply.
```
# first valid index for each column
df.apply(pd.Series.first_valid_index)

A    1
B    0
dtype: int64

# last valid index for each column
df.apply(pd.Series.last_valid_index)

A    3
B    0
dtype: int64
```
As before, you can also use notna and idxmax. This is slightly more natural syntax.
```
# first valid index
df.notna().idxmax()

A    1
B    0
dtype: int64

# last valid index
df.notna()[::-1].idxmax()

A    3
B    0
dtype: int64
```


Question:
What are the differences between feather and parquet? Both are columnar (disk-)storage formats for use in data analysis systems. Both are integrated within Apache Arrow (pyarrow package for python) and are designed to correspond with Arrow as a columnar in-memory analytics layer. How do both formats differ? Should you always prefer feather when working with pandas when possible? What are the use cases where feather is more suitable than parquet and the other way round? Appendix I found some hints here https://github.com/wesm/feather/issues/188, but given the young age of this project, it's possibly a bit out of date. Not a serious speed test because I'm just dumping and loading a whole Dataframe but to give you some impression if you never heard of the formats before:
```
# IPython
import numpy as np
import pandas as pd
import pyarrow as pa
import pyarrow.feather as feather
import pyarrow.parquet as pq
import fastparquet as fp

df = pd.DataFrame({'one': [-1, np.nan, 2.5],
                   'two': ['foo', 'bar', 'baz'],
                   'three': [True, False, True]})

print("pandas df to disk ####################################################")
print('example_feather:')
%timeit feather.write_feather(df, 'example_feather')
# 2.62 ms ± 35.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
print('example_parquet:')
%timeit pq.write_table(pa.Table.from_pandas(df), 'example.parquet')
# 3.19 ms ± 51 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
print()

print("for comparison:")
print('example_pickle:')
%timeit df.to_pickle('example_pickle')
# 2.75 ms ± 18.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
print('example_fp_parquet:')
%timeit fp.write('example_fp_parquet', df)
# 7.06 ms ± 205 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
print('example_hdf:')
%timeit df.to_hdf('example_hdf', 'key_to_store', mode='w', table=True)
# 24.6 ms ± 4.45 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
print()

print("pandas df from disk ##################################################")
print('example_feather:')
%timeit feather.read_feather('example_feather')
# 969 µs ± 1.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
print('example_parquet:')
%timeit pq.read_table('example.parquet').to_pandas()
# 1.9 ms ± 5.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

print("for comparison:")
print('example_pickle:')
%timeit pd.read_pickle('example_pickle')
# 1.07 ms ± 6.21 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
print('example_fp_parquet:')
%timeit fp.ParquetFile('example_fp_parquet').to_pandas()
# 4.53 ms ± 260 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
print('example_hdf:')
%timeit pd.read_hdf('example_hdf')
# 10 ms ± 43.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

# pandas version: 0.22.0
# fastparquet version: 0.1.3
# numpy version: 1.13.3
# pandas version: 0.22.0
# pyarrow version: 0.8.0
# sys.version: 3.6.3
# example Dataframe taken from https://arrow.apache.org/docs/python/parquet.html
```

Response: Parquet format is designed for long-term storage, where Arrow is more intended for short term or ephemeral storage (Arrow may be more suitable for long-term storage after the 1.0.0 release happens, since the binary format will be stable then) Parquet is more expensive to write than Feather as it features more layers of encoding and compression. Feather is unmodified raw columnar Arrow memory. We will probably add simple compression to Feather in the future. Due to dictionary encoding, RLE encoding, and data page compression, Parquet files will often be much smaller than Feather files Parquet is a standard storage format for analytics that's supported by many different systems: Spark, Hive, Impala, various AWS services, in future by BigQuery, etc. So if you are doing analytics, Parquet is a good option as a reference storage format for query by multiple systems The benchmarks you showed are going to be very noisy since the data you read and wrote is very small. You should try compressing at least 100MB or upwards 1GB of data to get some more informative benchmarks, see e.g. http://wesmckinney.com/blog/python-parquet-multithreading/


Question:
Pandas query function not working with spaces in column names I have a dataframe with spaces in column names. I am trying to use query method to get the results. It is working fine with 'c' column but getting error for 'a b'
```
import pandas as pd
a = pd.DataFrame(columns=["a b", "c"])
a["a b"] = [1,2,3,4]
a["c"] = [5,6,7,8]
a.query('a b==5')
```
For this I am getting this error:
```
a b ==5
  ^
SyntaxError: invalid syntax
```
I don't want to fill up space with other characters like '_' etc. There is one hack using pandasql to put variable name inside brackets example: [a b]

Response:
From pandas 0.25 onward you will be able to escape column names with backticks so you can do
```py
a.query('`a b` == 5')
```


Question: Is there a performance difference between Numpy and Pandas? I've written a bunch of code on the assumption that I was going to use Numpy arrays. Turns out the data I am getting is loaded through Pandas. I remember now that I loaded it in Pandas because I was having some problems loading it in Numpy. I believe the data was just too large. Therefore I was wondering, is there a difference in computational ability when using Numpy vs Pandas? If Pandas is more efficient then I would rather rewrite all my code for Pandas but if there is no more efficiency then I'll just use a numpy array...

Response: There can be a significant performance difference, of an order of magnitude for multiplications and multiple orders of magnitude for indexing a few random values. I was actually wondering about the same thing and came across this interesting comparison: http://penandpants.com/2014/09/05/performance-of-pandas-series-vs-numpy-arrays/


Question:
Using Merge on a column and Index in Pandas I have two separate dataframes that share a project number. In type_df, the project number is the index. In time_df, the project number is a column. I would like to count the number of rows in type_df that have a Project Type of 2. I am trying to do this with pandas.merge(). It works great when using both columns, but not indices. I'm not sure how to reference the index and if merge is even the right way to do this.
```
import pandas as pd
type_df = pd.DataFrame(data = [['Type 1'], ['Type 2']],
                       columns=['Project Type'],
                       index=['Project2', 'Project1'])
time_df = pd.DataFrame(data = [['Project1', 13], ['Project1', 12],
                               ['Project2', 41]],
                       columns=['Project', 'Time'])
merged = pd.merge(time_df,type_df, on=[index,'Project'])
print merged[merged['Project Type'] == 'Type 2']['Project Type'].count()
```
Error: Name 'Index' is not defined. Desired Output:
```
2
```

Response:
If you want to use an index in your merge you have to specify left_index=True or right_index=True, and then use left_on or right_on. For you it should look something like this:
```
merged = pd.merge(type_df, time_df, left_index=True, right_on='Project')
```


Question:
Convert column of date objects in Pandas DataFrame to strings How to convert a column consisting of datetime64 objects to a strings that would read 01-11-2013 for today's date of November 1. I have tried
```
df['DateStr'] = df['DateObj'].strftime('%d%m%Y')
```
but I get this error AttributeError: 'Series' object has no attribute 'strftime'

Response:
As of version 17.0, you can format with the dt accessor:
```
df['DateStr'] = df['DateObj'].dt.strftime('%d%m%Y')
```


Question:
Rename specific column(s) in pandas I've got a dataframe called data. How would I rename the only one column header? For example gdp to log(gdp)?
```
data =
    y  gdp  cap
0   1    2    5
1   2    3    9
2   8    7    2
3   3    4    7
4   6    7    7
5   4    8    3
6   8    2    8
7   9    9   10
8   6    6    4
9  10   10    7
```

Response:
```
data.rename(columns={'gdp':'log(gdp)'}, inplace=True)
```
The rename show that it accepts a dict as a param for columns so you just pass a dict with a single entry. Also see related


Question:
Converting between datetime, Timestamp and datetime64 How do I convert a numpy.datetime64 object to a datetime.datetime (or Timestamp)? In the following code, I create a datetime, timestamp and datetime64 objects.
```
import datetime
import numpy as np
import pandas as pd
dt = datetime.datetime(2012, 5, 1)
# A strange way to extract a Timestamp object, there's surely a better way?
ts = pd.DatetimeIndex([dt])[0]
dt64 = np.datetime64(dt)

```python
dt
#Output
#datetime.datetime(2012, 5, 1, 0, 0)
```

```python
ts
#Output
#<Timestamp: 2012-05-01 00:00:00>
```

```python
dt64
#Output
#numpy.datetime64('2012-05-01T01:00:00.000000+0100')
#```
#Note: it's easy to get the datetime from the Timestamp:
#```
```

```python
ts.to_datetime()
#Output
#datetime.datetime(2012, 5, 1, 0, 0)
#```
#But how do we extract the datetime or Timestamp from a numpy.datetime64 (dt64)? . Update: a somewhat nasty example in my dataset (perhaps the motivating example) seems to be:
#```
#dt64 = numpy.datetime64('2002-06-28T01:00:00.000000000+0100')
#```
#which should be datetime.datetime(2002, 6, 28, 1, 0), and not a long (!) (1025222400000000000L)...
```

Response: You can just use the pd.Timestamp constructor. The following diagram may be useful for this and related questions.


Question:
pandas comparison raises TypeError: cannot compare a dtyped [float64] array with a scalar of type [bool] I have the following structure to my dataFrame:
```
Index: 1008 entries, Trial1.0 to Trial3.84
Data columns (total 5 columns):
CHUNK_NAME                    1008  non-null values
LAMBDA                        1008  non-null values
BETA                          1008  non-null values
HIT_RATE                      1008  non-null values
AVERAGE_RECIPROCAL_HITRATE    1008  non-null values

chunks=['300_321','322_343','344_365','366_387','388_408','366_408','344_408','322_408','300_408']
lam_beta=[(lambda1,beta1),(lambda1,beta2),(lambda1,beta3),...(lambda1,beta_n),(lambda2,beta1),(lambda2,beta2)...(lambda2,beta_n),........]

my_df.ix[my_df.CHUNK_NAME==chunks[0]&my_df.LAMBDA==lam_beta[0][0]]
```
I want to get the rows of the DataFrame for a particular chunk lets say chunks[0] and particular lambda value. So in this case, the output should be all rows in the DataFrame having CHUNK_NAME='300_321' and LAMBDA=lambda1. There would be n rows one for each beta value that would be returned. But instead I get the following error. Any help in solving this problem would be appreciated.
```
TypeError: cannot compare a dtyped [float64] array with a scalar of type [bool]
```

Response:
& has higher precedence than ==. Write:
```
my_df.ix[(my_df.CHUNK_NAME==chunks[0])&(my_df.LAMBDA==lam_beta[0][0])]
         ^                           ^ ^                            ^
```


Question: How to proceed with `None` value in pandas fillna I have the following dictionary: fillna(value={'first_name':'Andrii', 'last_name':'Furmanets', 'created_at':None}) When I pass that dictionary to fillna I see: raise ValueError('must specify a fill method or value')\nValueError: must specify a fill method or value\n" It seems to me that it fails on None value. I use pandas version 0.20.3.

Response:
In case you want to normalize all of the nulls with python's None.
```
df.fillna(np.nan).replace([np.nan], [None])
```
The first fillna will replace all of (None, NAT, np.nan, etc) with Numpy's NaN, then replace Numpy's NaN with python's None.


Question:
Remove index column while saving csv in pandas I'm trying to create a csv with pandas, but when I export the data to csv it gives me an extra column
```py
d = {'one' : pd.Series([1., 2., 3.]),'two' : pd.Series([1., 2., 3., 4.])}
df0_fa = pd.DataFrame(d)
df_csv = df0_fa.to_csv('revenue/data/test.csv',mode = 'w')
```
Thus, my result is:
```
,one,two
0,1.0,1.0
1,2.0,2.0
2,3.0,3.0
3,4.0,4.0
```
But, the expected results are:
```
one,two
1.0,1.0
2.0,2.0
3.0,3.0
4.0,4.0
```

Response:
What you are seeing is the index column. Just set index=False:
```
df_csv = df0_fa.to_csv('revenue/data/test.csv',mode = 'w', index=False)
```


Question:
How to plot different groups of data from a dataframe into a single figure I have a temperature file with many years of temperature records in the format below:
```none
2012-04-12,16:13:09,20.6
2012-04-12,17:13:09,20.9
2012-04-12,18:13:09,20.6
2007-05-12,19:13:09,5.4
2007-05-12,20:13:09,20.6
2007-05-12,20:13:09,20.6
2005-08-11,11:13:09,20.6
2005-08-11,11:13:09,17.5
2005-08-13,07:13:09,20.6
2006-04-13,01:13:09,20.6
```
Every year has different numbers of time of records, so the pandas datetimeindices are all different. I want to plot the different year's data in the same figure for comparison: The X-axis is datetimeindices from Jan to Dec The Y-axis is the temperature How should I go about doing this?

Response:
Try:
```
ax = df1.plot()
df2.plot(ax=ax)
```


Question:
Add a string prefix to each value in a pandas string column I would like to prepend a string to the start of each value in a said column of a pandas dataframe. I am currently using:
```py
df.ix[(df['col'] != False), 'col'] = 'str' + df[(df['col'] != False), 'col']
```
This seems an inelegant method. Do you know any other way (which maybe also adds the character to rows where that column is 0 or NaN)? As an example, I would like to turn:
```none
col
1     a
2     0
```
into:
```none
col
1     stra
2     str0
```

Response:
```
df['col'] = 'str' + df['col'].astype(str)
```
Example:
```

```python
df = pd.DataFrame({'col':['a',0]})
```

```python
df
#Output
#  col
#0   a
#1   0
```

```python
df['col'] = 'str' + df['col'].astype(str)
```

```python
df
#Output
#    col
#0  stra
#1  str0
#```
```


Question:
Renaming Column Names in Pandas Groupby function [duplicate] This question already has answers here: Multiple aggregations of the same column using pandas GroupBy.agg() (5 answers) Closed 6 years ago. Q1) I want to do a groupby, SQL-style aggregation and rename the output column: Example dataset:
```
ID     Region  count
0  100       Asia      2
1  101     Europe      3
2  102         US      1
3  103     Africa      5
4  100     Russia      5
5  101  Australia      7
6  102         US      8
7  104       Asia     10
8  105     Europe     11
9  110     Africa     23
```
I want to group the observations of this dataset by ID and Region and summing the count for each group. So I used something like this...
```

```python
print(df.groupby(['ID','Region'],as_index=False).count().sum())
```

ID     Region  count
0  100       Asia      2
1  100     Russia      5
2  101  Australia      7
3  101     Europe      3
4  102         US      9
5  103     Africa      5
6  104       Asia     10
7  105     Europe     11
8  110     Africa     23
```
On using as_index=False I am able to get "SQL-Like" output. My problem is that I am unable to rename the aggregate variable count here. So in SQL if wanted to do the above thing I would do something like this:
```
select ID, Region, sum(count) as Total_Numbers
from df
group by ID, Region
order by ID, Region
```
As we see, it's very easy for me to rename the aggregate variable count to Total_Numbers in SQL. I wanted to do the same thing in Pandas but unable to find such an option in group-by function. Can somebody help? The second question (more of an observation): Q2) Is it possible to directly use column names in Pandas dataframe functions without enclosing them in quotes? I understand that the variable names are strings, so have to be inside quotes, but I see if use them outside dataframe function and as an attribute we don't require them to be inside quotes. Like df.ID.sum() etc. It's only when we use it in a DataFrame function like df.sort() or df.groupby we have to use it inside quotes. This is actually a bit of pain as in SQL or in SAS or other languages we simply use the variable name without quoting them. Any suggestion on this? Kindly reply to both questions (Q1 is the main, Q2 more of an opinion).

Response:
For the first question I think answer would be:
```
<your DataFrame>.rename(columns= {'count':'Total_Numbers'})
```
or
```
<your DataFrame>.columns = ['ID', 'Region', 'Total_Numbers']
```
if it is Pandas Series we have
```
<your Series>.rename("Total_Numbers")
```
As for second one I'd say the answer would be no. It's possible to use it like 'df.ID' because of python datamodel: Attribute references are translated to lookups in this dictionary, e.g., m.x is equivalent to m.dict["x"]


Question:
Efficiently select rows that match one of several values in Pandas DataFrame [duplicate] This question already has answers here: Use a list of values to select rows from a Pandas dataframe (9 answers) Closed 3 years ago. Problem Given data in a Pandas DataFrame like the following:
```
Name     Amount
---------------
Alice       100
Bob          50
Charlie     200
Alice        30
Charlie      10
```
I want to select all rows where the Name is one of several values in a collection {Alice, Bob}
```
Name     Amount
---------------
Alice       100
Bob          50
Alice        30
```
Question What is an efficient way to do this in Pandas? Options as I see them Loop through rows, handling the logic with Python Select and merge many statements like the following
```
merge(df[df.name = specific_name] for specific_name in names) # something like this
```
Perform some sort of join What are the performance trade-offs here? When is one solution better than the others? What solutions am I missing? While the example above uses strings my actual job uses matches on 10-100 integers over millions of rows and so fast NumPy operations may be relevant.

Response:
You can use the isin Series method:
```

```python
df['Name'].isin(['Alice', 'Bob'])
#Output
#0     True
#1     True
#2    False
#3     True
#4    False
#Name: Name, dtype: bool
```

```python
df[df.Name.isin(['Alice', 'Bob'])]
#Output
#Name  Amount
#0  Alice     100
#1    Bob      50
#3  Alice      30
#```
```


Question:
Pandas: Shift down values by one row within a group I have a Pandas dataframe, and I want to create a new column whose values are that of another column, shifted down by one row. The last row should show NaN. The catch is that I want to do this by group, with the last row of each group showing NaN. NOT have the last row of a group "steal" a value from a group that happens to be adjacent in the dataframe. My attempted implementation is quite shamefully broken, so I'm clearly misunderstanding something fundamental.
```
df['B_shifted'] = df.groupby(['A'])['B'].transform(lambda x:x.values[1:])
```

Response:
Newer versions of pandas can now perform a shift on a group:
```
df['B_shifted'] = df.groupby(['A'])['B'].shift(1)
```
Note that when shifting down, it's the first row that has NaN.


Question:
Get first letter of a string from column I'm fighting with pandas and for now I'm loosing. I have source table similar to this:
```
import pandas as pd

a=pd.Series([123,22,32,453,45,453,56])
b=pd.Series([234,4353,355,453,345,453,56])
df=pd.concat([a, b], axis=1)
df.columns=['First', 'Second']
```
I would like to add new column to this data frame with first digit from values in column 'First': a) change number to string from column 'First' b) extracting first character from newly created string c) Results from b save as new column in data frame I don't know how to apply this to the pandas data frame object. I would be grateful for helping me with that.

Response:
Cast the dtype of the col to str and you can perform vectorised slicing calling str:
```

```python
df['new_col'] = df['First'].astype(str).str[0]
df
#Output
#First  Second new_col
#0    123     234       1
#1     22    4353       2
#2     32     355       3
#3    453     453       4
#4     45     345       4
#5    453     453       4
#6     56      56       5
#```
#if you need to you can cast the dtype back again calling astype(int) on the column
```


Question:
How to check if particular value (in cell) is NaN in pandas DataFrame? Let's say I have the following pandas DataFrame:
```
import pandas as pd
import numpy as np
df = pd.DataFrame({"A": [1, np.nan, 2], "B": [5, 6, 0]})
```
Which would look like:
```

```python
df
#Output
#     A  B
#0  1.0  5
#1  NaN  6
#2  2.0  0
#```
#First option I know one way to check if a particular value is NaN:
#```
```

```python
df.isnull().iloc[1,0]
#Output
#True
#```
#But this checks the whole dataframe just to get one value, so I imagine it's wasteful. Second option (not working) I thought below option, using iloc, would work as well, but it doesn't:
#```
```

```python
df.iloc[1,0] == np.nan
#Output
#False
#```
#However if I check that value I get:
#```
```

```python
df.iloc[1,0]
#Output
#nan
#```
#So, why is the second option not working? Is it possible to check for NaN values using iloc? Editor's note: This question previously used pd.np instead of np and .ix in addition to .iloc, but since these no longer exist, they have been edited out to keep it short and clear.
```

Response:
Try pd.isna():
```

```python
pd.isna(df.iloc[1,0])
#Output
#True
#```
#AKA pd.isnull
```


Question: How to change the datetime format in Pandas My dataframe has a DOB column (example format 1/26/2016) which by default gets converted to Pandas dtype 'object'. Converting this to date format with df['DOB'] = pd.to_datetime(df['DOB']), the date gets converted to: 2016-01-26 and its dtype is datetime64[ns]. Now I want to convert this date format to 01/26/2016 or any other general date format. How do I do it? (Whatever the method I try, it always shows the date in 2016-01-26 format.)

Response:
You can use dt.strftime if you need to convert datetime to other formats (but note that then dtype of column will be object (string)):
```
import pandas as pd

df = pd.DataFrame({'DOB': {0: '26/1/2016', 1: '26/1/2016'}})
print (df)
         DOB
0  26/1/2016
1  26/1/2016

df['DOB'] = pd.to_datetime(df.DOB)
print (df)
         DOB
0 2016-01-26
1 2016-01-26

df['DOB1'] = df['DOB'].dt.strftime('%m/%d/%Y')
print (df)
         DOB        DOB1
0 2016-01-26  01/26/2016
1 2016-01-26  01/26/2016
```


Question:
How to write to an existing excel file without overwriting data (using pandas)? I use pandas to write to excel file in the following fashion:
```
import pandas

writer = pandas.ExcelWriter('Masterfile.xlsx')

data_filtered.to_excel(writer, "Main", cols=['Diff1', 'Diff2'])

writer.save()
```
Masterfile.xlsx already consists of number of different tabs. However, it does not yet contain "Main". Pandas correctly writes to "Main" sheet, unfortunately it also deletes all other tabs.

Response:
Pandas docs says it uses openpyxl for xlsx files. Quick look through the code in ExcelWriter gives a clue that something like this might work out:
```
import pandas
from openpyxl import load_workbook

book = load_workbook('Masterfile.xlsx')
writer = pandas.ExcelWriter('Masterfile.xlsx', engine='openpyxl')
writer.book = book

## ExcelWriter for some reason uses writer.sheets to access the sheet.
## If you leave it empty it will not know that sheet Main is already there
## and will create a new sheet.

writer.sheets = dict((ws.title, ws) for ws in book.worksheets)

data_filtered.to_excel(writer, "Main", cols=['Diff1', 'Diff2'])

writer.save()
```


Question:
Logical operators for Boolean indexing in Pandas I'm working with a Boolean index in Pandas. The question is why the statement:
```
a[(a['some_column']==some_number) & (a['some_other_column']==some_other_number)]
```
works fine whereas
```
a[(a['some_column']==some_number) and (a['some_other_column']==some_other_number)]
```
exits with error? Example:
```
a = pd.DataFrame({'x':[1,1],'y':[10,20]})

In: a[(a['x']==1)&(a['y']==10)]
Out:    x   y
     0  1  10

In: a[(a['x']==1) and (a['y']==10)]
Out: ValueError: The truth value of an array with more than one element is ambiguous.     Use a.any() or a.all()
```

Response:
When you say
```
(a['x']==1) and (a['y']==10)
```
You are implicitly asking Python to convert (a['x']==1) and (a['y']==10) to Boolean values. NumPy arrays (of length greater than 1) and Pandas objects such as Series do not have a Boolean value -- in other words, they raise ValueError: The truth value of an array is ambiguous. Use a.empty, a.any() or a.all(). when used as a Boolean value. That's because it's unclear when it should be True or False. Some users might assume they are True if they have non-zero length, like a Python list. Others might desire for it to be True only if all its elements are True. Others might want it to be True if any of its elements are True. Because there are so many conflicting expectations, the designers of NumPy and Pandas refuse to guess, and instead raise a ValueError. Instead, you must be explicit, by calling the empty(), all() or any() method to indicate which behavior you desire. In this case, however, it looks like you do not want Boolean evaluation, you want element-wise logical-and. That is what the & binary operator performs:
```
(a['x']==1) & (a['y']==10)
```
returns a boolean array. By the way, as alexpmil notes, the parentheses are mandatory since & has a higher operator precedence than ==. Without the parentheses,
```
a['x']==1 & a['y']==10
```
would be evaluated as
```
a['x'] == (1 & a['y']) == 10
```
which would in turn be equivalent to the chained comparison
```
(a['x'] == (1 & a['y'])) and ((1 & a['y']) == 10)
```
That is an expression of the form Series and Series. The use of and with two Series would again trigger the same ValueError as above. That's why the parentheses are mandatory.


Question:
Finding common rows (intersection) in two Pandas dataframes Assume I have two dataframes of this format (call them df1 and df2):
```
+------------------------+------------------------+--------+
|        user_id         |      business_id       | rating |
+------------------------+------------------------+--------+
| rLtl8ZkDX5vH5nAx9C3q5Q | eIxSLxzIlfExI6vgAbn2JA |      4 |
| C6IOtaaYdLIT5fWd7ZYIuA | eIxSLxzIlfExI6vgAbn2JA |      5 |
| mlBC3pN9GXlUUfQi1qBBZA | KoIRdcIfh3XWxiCeV1BDmA |      3 |
+------------------------+------------------------+--------+
```
I'm looking to get a dataframe of all the rows that have a common user_id in df1 and df2. (ie. if a user_id is in both df1 and df2, include the two rows in the output dataframe) I can think of many ways to approach this, but they all strike me as clunky. For example, we could find all the unique user_ids in each dataframe, create a set of each, find their intersection, filter the two dataframes with the resulting set and concatenate the two filtered dataframes. Maybe that's the best approach, but I know Pandas is clever. Is there a simpler way to do this? I've looked at merge but I don't think that's what I need.

Response:
My understanding is that this question is better answered over in this post. But briefly, the answer to the OP with this method is simply:
```
s1 = pd.merge(df1, df2, how='inner', on=['user_id'])
```
Which gives s1 with 5 columns: user_id and the other two columns from each of df1 and df2.


Question:
How to form tuple column from two columns in Pandas I've got a Pandas DataFrame and I want to combine the 'lat' and 'long' columns to form a tuple.
```
<class 'pandas.core.frame.DataFrame'>
Int64Index: 205482 entries, 0 to 209018
Data columns:
Month           205482  non-null values
Reported by     205482  non-null values
Falls within    205482  non-null values
Easting         205482  non-null values
Northing        205482  non-null values
Location        205482  non-null values
Crime type      205482  non-null values
long            205482  non-null values
lat             205482  non-null values
dtypes: float64(4), object(5)
```
The code I tried to use was:
```
def merge_two_cols(series):
    return (series['lat'], series['long'])

sample['lat_long'] = sample.apply(merge_two_cols, axis=1)
```
However, this returned the following error:
```
---------------------------------------------------------------------------
 AssertionError                            Traceback (most recent call last)
<ipython-input-261-e752e52a96e6> in <module>()
      2     return (series['lat'], series['long'])
      3
----> 4 sample['lat_long'] = sample.apply(merge_two_cols, axis=1)
      5
```
...
```
AssertionError: Block shape incompatible with manager
```
How can I solve this problem?

Response:
Get comfortable with zip. It comes in handy when dealing with column data.
```
df['new_col'] = list(zip(df.lat, df.long))
```
It's less complicated and faster than using apply or map. Something like np.dstack is twice as fast as zip, but wouldn't give you tuples.


Question:
How to select rows with NaN in particular column? Given this dataframe, how to select only those rows that have "Col2" equal to NaN?
```
df = pd.DataFrame([range(3), [0, np.NaN, 0], [0, 0, np.NaN], range(3), range(3)], columns=["Col1", "Col2", "Col3"])
```
which looks like:
```
0   1   2
0  0   1   2
1  0 NaN   0
2  0   0 NaN
3  0   1   2
4  0   1   2
```
The result should be this one:
```
0   1   2
1  0 NaN   0
```

Response:
Try the following:
```
df[df['Col2'].isnull()]
```


Question: Split pandas dataframe in two if it has more than 10 rows I have a huge CSV with many tables with many rows. I would like to simply split each dataframe into 2 if it contains more than 10 rows. If true, I would like the first dataframe to contain the first 10 and the rest in the second dataframe. Is there a convenient function for this? I've looked around but found nothing useful... i.e. split_dataframe(df, 2(if > 10))?

Response:
I used a List Comprehension to cut a huge DataFrame into blocks of 100'000:
```
size = 100000
list_of_dfs = [df.loc[i:i+size-1,:] for i in range(0, len(df),size)]
```
or as generator:
```
list_of_dfs = (df.loc[i:i+size-1,:] for i in range(0, len(df),size))
```


Question:
Python pandas equivalent to R groupby mutate So in R when I have a data frame consisting of say 4 columns, call it df and I want to compute the ratio by sum product of a group, I can it in such a way:
```
// generate data
df = data.frame(a=c(1,1,0,1,0),b=c(1,0,0,1,0),c=c(10,5,1,5,10),d=c(3,1,2,1,2));
| a   b   c    d |
| 1   1   10   3 |
| 1   0   5    1 |
| 0   0   1    2 |
| 1   1   5    1 |
| 0   0   10   2 |
// compute sum product ratio
df = df%>% group_by(a,b) %>%
      mutate(
          ratio=c/sum(c*d)
      );
| a   b   c    d  ratio |
| 1   1   10   3  0.286 |
| 1   1   5    1  0.143 |
| 1   0   5    1  1     |
| 0   0   1    2  0.045 |
| 0   0   10   2  0.454 |
```
But in python I need to resort to loops. I know there should be a more elegant way than raw loops in python, anyone got any ideas?

Response:
It can be done with similar syntax with groupby() and apply():
```
df['ratio'] = df.groupby(['a','b'], group_keys=False).apply(lambda g: g.c/(g.c * g.d).sum())
```


Question:
How to flatten multilevel/nested JSON? I am trying to convert JSON to CSV file, that I can use for further analysis. Issue with my structure is that I have quite some nested dict/lists when I convert my JSON file. I tried to use pandas json_normalize(), but it only flattens first level.
```
import json
import pandas as pd
from pandas.io.json import json_normalize
from cs import CloudStack

api_key = xxxx
secret = xxxx
endpoint = xxxx

cs = CloudStack(endpoint=endpoint,
                key=api_key,
                secret=secret)

virtual_machines = cs.virtMach()

test = json_normalize(virtual_machines["virtualmachine"])

test.to_csv("test.csv", sep="|", index=False)
```
Any idea how to flatter whole JSON file, so I can create single line input to CSV file for single (in this case virtual machine) entry? I have tried couple of solutions posted here, but my result was always only first level was flattened. This is sample JSON (in this case, I still get "securitygroup" and "nic" output as JSON format:
```
{
    "count": 13,
    "virtualmachine": [
        {
            "id": "1082e2ed-ff66-40b1-a41b-26061afd4a0b",
            "name": "test-2",
            "displayname": "test-2",
            "securitygroup": [
                {
                    "id": "9e649fbc-3e64-4395-9629-5e1215b34e58",
                    "name": "test",
                    "tags": []
                }
            ],
            "nic": [
                {
                    "id": "79568b14-b377-4d4f-b024-87dc22492b8e",
                    "networkid": "05c0e278-7ab4-4a6d-aa9c-3158620b6471"
                },
                {
                    "id": "3d7f2818-1f19-46e7-aa98-956526c5b1ad",
                    "networkid": "b4648cfd-0795-43fc-9e50-6ee9ddefc5bd",
                    "traffictype": "Guest"
                }
            ],
            "hypervisor": "KVM",
            "affinitygroup": [],
            "isdynamicallyscalable": false
        }
    ]
}
```

Response:
I used the following function (details can be found here):
```
def flatten_data(y):
    out = {}

    def flatten(x, name=''):
        if type(x) is dict:
            for a in x:
                flatten(x[a], name + a + '_')
        elif type(x) is list:
            i = 0
            for a in x:
                flatten(a, name + str(i) + '_')
                i += 1
        else:
            out[name[:-1]] = x

    flatten(y)
    return out
```
This unfortunately completely flattens whole JSON, meaning that if you have multi-level JSON (many nested dictionaries), it might flatten everything into single line with tons of columns. What I used, in the end, was json_normalize() and specified structure that I required. A nice example of how to do it that way can be found here.


Question: What is the difference between size and count in pandas? That is the difference between groupby("x").count and groupby("x").size in pandas ? Does size just exclude nil ?

Response:
size includes NaN values, count does not:
```

```python
df = pd.DataFrame({'a':[0,0,1,2,2,2], 'b':[1,2,3,4,np.NaN,4], 'c':np.random.randn(6)})
df
#Output
#a   b         c
#0  0   1  1.067627
#1  0   2  0.554691
#2  1   3  0.458084
#3  2   4  0.426635
#4  2 NaN -2.238091
#5  2   4  1.256943
```

```python
print(df.groupby(['a'])['b'].count())
print(df.groupby(['a'])['b'].size())

a
0    2
1    1
2    2
Name: b, dtype: int64

a
0    2
1    1
2    3
dtype: int64
```
```


Question:
Use variable in Pandas query I'm trying to query a Pandas dataframe like this:
```
inv = pd.read_csv(infile)
inv.columns = ['County', 'Site', 'Role', 'Hostname']
clist = inv.County.unique()  # Get list of counties
for county in clist:  # for each county
    csub = inv.query('County == county')  # create a county subset
    # ... do stuff on subset
```
But I get an error:
```
pandas.core.computation.ops.UndefinedVariableError: name 'county' is not defined
```
I'm sure it's a trivial error, but I can't figure it out. How do I pass a variable to the query method?

Response:
According to the documentation, you can reference variables using @:
```
csub = inv.query('County == @county')
```


Question:
Groupby value counts on the dataframe pandas I have the following dataframe:
```
df = pd.DataFrame([
    (1, 1, 'term1'),
    (1, 2, 'term2'),
    (1, 1, 'term1'),
    (1, 1, 'term2'),
    (2, 2, 'term3'),
    (2, 3, 'term1'),
    (2, 2, 'term1')
], columns=['id', 'group', 'term'])
```
I want to group it by id and group and calculate the number of each term for this id-group pair. So in the end I want to get something like this: Anyway I can achieve this without looping?

Response:
I use groupby and size
```
df.groupby(['id', 'group', 'term']).size().unstack(fill_value=0)
```
Timing 1,000,000 rows
```
df = pd.DataFrame(dict(id=np.random.choice(100, 1000000),
                       group=np.random.choice(20, 1000000),
                       term=np.random.choice(10, 1000000)))
```


Question:
What is dtype('O'), in pandas? I have a dataframe in pandas and I'm trying to figure out what the types of its values are. I am unsure what the type is of column 'Test'. However, when I run myFrame['Test'].dtype, I get;
```
dtype('O')
```
What does this mean?

Response:
It means:
```
'O'     (Python) objects
```
Source. The first character specifies the kind of data and the remaining characters specify the number of bytes per item, except for Unicode, where it is interpreted as the number of characters. The item size must correspond to an existing type, or an error will be raised. The supported kinds are to an existing type, or an error will be raised. The supported kinds are:
```
'b'       boolean
'i'       (signed) integer
'u'       unsigned integer
'f'       floating-point
'c'       complex-floating point
'O'       (Python) objects
'S', 'a'  (byte-)string
'U'       Unicode
'V'       raw data (void)
```
Another answer helps if need check types.


Question:
How to iterate over consecutive chunks of Pandas dataframe efficiently I have a large dataframe (several million rows). I want to be able to do a groupby operation on it, but just grouping by arbitrary consecutive (preferably equal-sized) subsets of rows, rather than using any particular property of the individual rows to decide which group they go to. The use case: I want to apply a function to each row via a parallel map in IPython. It doesn't matter which rows go to which back-end engine, as the function calculates a result based on one row at a time. (Conceptually at least; in reality it's vectorized.) I've come up with something like this:
```
# Generate a number from 0-9 for each row, indicating which tenth of the DF it belongs to
max_idx = dataframe.index.max()
tenths = ((10 * dataframe.index) / (1 + max_idx)).astype(np.uint32)

# Use this value to perform a groupby, yielding 10 consecutive chunks
groups = [g[1] for g in dataframe.groupby(tenths)]

# Process chunks in parallel
results = dview.map_sync(my_function, groups)
```
But this seems very long-winded, and doesn't guarantee equal sized chunks. Especially if the index is sparse or non-integer or whatever. Any suggestions for a better way? Thanks!

Response:
Use numpy's array_split():
```
import numpy as np
import pandas as pd

data = pd.DataFrame(np.random.rand(10, 3))
for chunk in np.array_split(data, 5):
  assert len(chunk) == len(data) / 5, "This assert may fail for the last chunk if data lenght isn't divisible by 5"
```


Question:
Cumsum as a new column in an existing Pandas dataframe I have a pandas dataframe defined as:
```
A   B   SUM_C
1   1   10
1   2   20
```
I would like to do a cumulative sum of SUM_C and add it as a new column to the same dataframe. In other words, my end goal is to have a dataframe that looks like below:
```
A   B   SUM_C   CUMSUM_C
1   1   10      10
1   2   20      30
```
Using cumsum in pandas on group() shows the possibility of generating a new dataframe where column name SUM_C is replaced with cumulative sum. However, my ask is to add the cumulative sum as a new column to the existing dataframe.

Response:
Just apply cumsum on the pandas.Series df['SUM_C'] and assign it to a new column:
```
df['CUMSUM_C'] = df['SUM_C'].cumsum()
```
Result:
```
df
Out[34]:
   A  B  SUM_C  CUMSUM_C
0  1  1     10       10
1  1  2     20       30
```


Question:
Python Pandas to_sql, how to create a table with a primary key? I would like to create a MySQL table with Pandas' to_sql function which has a primary key (it is usually kind of good to have a primary key in a mysql table) as so:
```
group_export.to_sql(con = db, name = config.table_group_export, if_exists = 'replace', flavor = 'mysql', index = False)
```
but this creates a table without any primary key, (or even without any index). The documentation mentions the parameter 'index_label' which combined with the 'index' parameter could be used to create an index but doesn't mention any option for primary keys. Documentation

Response:
Simply add the primary key after uploading the table with pandas.
```
group_export.to_sql(con=engine, name=example_table, if_exists='replace',
                    flavor='mysql', index=False)

with engine.connect() as con:
    con.execute('ALTER TABLE `example_table` ADD PRIMARY KEY (`ID_column`);')
```


Question:
How to deal with SettingWithCopyWarning in Pandas I just upgraded my Pandas from 0.11 to 0.13.0rc1. Now, the application is popping out many new warnings. One of them like this:
```none
E:\FinReporter\FM_EXT.py:449: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
  quote_df['TVol']   = quote_df['TVol']/TVOL_SCALE
```
What exactly does it mean? Do I need to change something? How should I suspend the warning if I insist on using quote_df['TVol'] = quote_df['TVol']/TVOL_SCALE? The function that gives warnings
```
def _decode_stock_quote(list_of_150_stk_str):
    """decode the webpage and return dataframe"""

    from cStringIO import StringIO

    str_of_all = "".join(list_of_150_stk_str)

    quote_df = pd.read_csv(
        StringIO(str_of_all),
        sep=',',
        names=list('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefg'))
        #dtype={'A': object, 'B': object, 'C': np.float64}
    quote_df.rename(
        columns={
            'A':'STK', 'B':'TOpen', 'C':'TPCLOSE', 'D':'TPrice',
            'E':'THigh', 'F':'TLow', 'I':'TVol', 'J':'TAmt',
            'e':'TDate', 'f':'TTime'},
        inplace=True)
    quote_df = quote_df.ix[:,[0,3,2,1,4,5,8,9,30,31]]
    quote_df['TClose'] = quote_df['TPrice']
    quote_df['RT']     = 100 * (quote_df['TPrice']/quote_df['TPCLOSE'] - 1)
    quote_df['TVol']   = quote_df['TVol']/TVOL_SCALE
    quote_df['TAmt']   = quote_df['TAmt']/TAMT_SCALE
    quote_df['STK_ID'] = quote_df['STK'].str.slice(13,19)
    quote_df['STK_Name'] = quote_df['STK'].str.slice(21,30)#.decode('gb2312')
    quote_df['TDate']  = quote_df.TDate.map(lambda x: x[0:4]+x[5:7]+x[8:10])

    return quote_df
```
More warning messages
```none
E:\FinReporter\FM_EXT.py:449: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
  quote_df['TVol']   = quote_df['TVol']/TVOL_SCALE
E:\FinReporter\FM_EXT.py:450: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
  quote_df['TAmt']   = quote_df['TAmt']/TAMT_SCALE
E:\FinReporter\FM_EXT.py:453: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
  quote_df['TDate']  = quote_df.TDate.map(lambda x: x[0:4]+x[5:7]+x[8:10])
```

Response:
The SettingWithCopyWarning was created to flag potentially confusing "chained" assignments, such as the following, which does not always work as expected, particularly when the first selection returns a copy. [see GH5390 and GH5597 for background discussion.]
```
df[df['A'] > 2]['B'] = new_val  # new_val not set in df
```
The warning offers a suggestion to rewrite as follows:
```
df.loc[df['A'] > 2, 'B'] = new_val
```
However, this doesn't fit your usage, which is equivalent to:
```
df = df[df['A'] > 2]
df['B'] = new_val
```
While it's clear that you don't care about writes making it back to the original frame (since you are overwriting the reference to it), unfortunately this pattern cannot be differentiated from the first chained assignment example. Hence the (false positive) warning. The potential for false positives is addressed in the docs on indexing, if you'd like to read further. You can safely disable this new warning with the following assignment.
```
import pandas as pd
pd.options.mode.chained_assignment = None  # default='warn'
```
Other Resources pandas User Guide: Indexing and selecting data Python Data Science Handbook: Data Indexing and Selection Real Python: SettingWithCopyWarning in Pandas: Views vs Copies Dataquest: SettingwithCopyWarning: How to Fix This Warning in Pandas Towards Data Science: Explaining the SettingWithCopyWarning in pandas


Question:
Import CSV file as a Pandas DataFrame How do I read the following CSV file into a Pandas DataFrame?
```
Date,"price","factor_1","factor_2"
2012-06-11,1600.20,1.255,1.548
2012-06-12,1610.02,1.258,1.554
2012-06-13,1618.07,1.249,1.552
2012-06-14,1624.40,1.253,1.556
2012-06-15,1626.15,1.258,1.552
2012-06-16,1626.15,1.263,1.558
2012-06-17,1626.15,1.264,1.572
```

Response:
pandas.read_csv to the rescue:
```
import pandas as pd
df = pd.read_csv("data.csv")
print(df)
```
This outputs a pandas DataFrame:
```
Date    price  factor_1  factor_2
0  2012-06-11  1600.20     1.255     1.548
1  2012-06-12  1610.02     1.258     1.554
2  2012-06-13  1618.07     1.249     1.552
3  2012-06-14  1624.40     1.253     1.556
4  2012-06-15  1626.15     1.258     1.552
5  2012-06-16  1626.15     1.263     1.558
6  2012-06-17  1626.15     1.264     1.572
```


Question:
How to filter rows containing a string pattern from a Pandas dataframe [duplicate] This question already has answers here: Filter pandas DataFrame by substring criteria (18 answers) Closed 6 years ago. Assume we have a data frame in Python Pandas that looks like this:
```
df = pd.DataFrame({'vals': [1, 2, 3, 4], 'ids': [u'aball', u'bball', u'cnut', u'fball']})
```
Or, in table form:
```none
ids    vals
aball   1
bball   2
cnut    3
fball   4
```
How do I filter rows which contain the key word "ball?" For example, the output should be:
```none
ids    vals
aball   1
bball   2
fball   4
```

Response:
```

```python
df[df['ids'].str.contains("ball")]
#Output
#ids  vals
#0  aball     1
#1  bball     2
#3  fball     4
#```
```


Question:
How to get value counts for multiple columns at once in Pandas DataFrame? Given a Pandas DataFrame that has multiple columns with categorical values (0 or 1), is it possible to conveniently get the value_counts for every column at the same time? For example, suppose I generate a DataFrame as follows:
```
import numpy as np
import pandas as pd
np.random.seed(0)
df = pd.DataFrame(np.random.randint(0, 2, (10, 4)), columns=list('abcd'))
```
I can get a DataFrame like this:
```
a  b  c  d
0  0  1  1  0
1  1  1  1  1
2  1  1  1  0
3  0  1  0  0
4  0  0  0  1
5  0  1  1  0
6  0  1  1  1
7  1  0  1  0
8  1  0  1  1
9  0  1  1  0
```
How do I conveniently get the value counts for every column and obtain the following conveniently?
```
a  b  c  d
0  6  3  2  6
1  4  7  8  4
```
My current solution is:
```
pieces = []
for col in df.columns:
    tmp_series = df[col].value_counts()
    tmp_series.name = col
    pieces.append(tmp_series)
df_value_counts = pd.concat(pieces, axis=1)
```
But there must be a simpler way, like stacking, pivoting, or groupby?

Response:
Just call apply and pass pd.Series.value_counts:
```

```python
df = pd.DataFrame(np.random.randint(0, 2, (10, 4)), columns=list('abcd'))
df.apply(pd.Series.value_counts)
#Output
#a  b  c  d
#0  4  6  4  3
#1  6  4  6  7
#```
```


Question:
pandas - find first occurrence Suppose I have a structured dataframe as follows:
```
df = pd.DataFrame({"A":['a','a','a','b','b'],
                   "B":[1]*5})
```
The A column has previously been sorted. I wish to find the first row index of where df[df.A!='a']. The end goal is to use this index to break the data frame into groups based on A. Now I realise that there is a groupby functionality. However, the dataframe is quite large and this is a simplified toy example. Since A has been sorted already, it would be faster if I can just find the 1st index of where df.A!='a'. Therefore it is important that whatever method that you use the scanning stops once the first element is found.

Response:
idxmax and argmax will return the position of the maximal value or the first position if the maximal value occurs more than once. use idxmax on df.A.ne('a')
```
df.A.ne('a').idxmax()

3
```
or the numpy equivalent
```
(df.A.values != 'a').argmax()

3
```
However, if A has already been sorted, then we can use searchsorted
```
df.A.searchsorted('a', side='right')

array([3])
```
Or the numpy equivalent
```
df.A.values.searchsorted('a', side='right')

3
```


Question:
Pandas: create two new columns in a dataframe with values calculated from a pre-existing column I am working with the pandas library and I want to add two new columns to a dataframe df with n columns (n > 0). These new columns result from the application of a function to one of the columns in the dataframe. The function to apply is like:
```
def calculate(x):
    ...operate...
    return z, y
```
One method for creating a new column for a function returning only a value is:
```
df['new_col']) = df['column_A'].map(a_function)
```
So, what I want, and tried unsuccesfully (*), is something like:
```
(df['new_col_zetas'], df['new_col_ys']) = df['column_A'].map(calculate)
```
What the best way to accomplish this could be ? I scanned the documentation with no clue. **df['column_A'].map(calculate) returns a pandas Series each item consisting of a tuple z, y. And trying to assign this to two dataframe columns produces a ValueError.*

Response:
I'd just use zip:
```

```python
from pandas import *
```

```python
def calculate(x):
   ...:     return x*2, x*3
   ...:
```

```python
df = DataFrame({'a': [1,2,3], 'b': [2,3,4]})
```

```python
df
#Output
#a  b
#0  1  2
#1  2  3
#2  3  4
```

```python
df["A1"], df["A2"] = zip(*df["a"].map(calculate))
```

```python
df
#Output
#a  b  A1  A2
#0  1  2   2   3
#1  2  3   4   6
#2  3  4   6   9
#```
```


Question:
Accessing Pandas column using squared brackets vs using a dot (like an attribute) In both the bellow cases:
```
import pandas

d = {'col1': 2, 'col2': 2.5}
df = pandas.DataFrame(data=d, index=[0])

print(df['col2'])
print(df.col2)
```
Both methods can be used to index on a column and yield the same result, so is there any difference between them?

Response: The "dot notation", i.e. df.col2 is the attribute access that's exposed as a convenience. You may access an index on a Series, column on a DataFrame, and an item on a Panel directly as an attribute: df['col2'] does the same: it returns a pd.Series of the column. A few caveats about attribute access: you cannot add a column (df.new_col = x won't work, worse: it will silently actually create a new attribute rather than a column - think monkey-patching here) it won't work if you have spaces in the column name or if the column name is an integer.


Question:
Extract values in Pandas value_counts() Say we have used pandas dataframe[column].value_counts() which outputs:
```
apple   5
 sausage 2
 banana  2
 cheese  1
```
How do you extract the values in the order same as shown above from max to min ? e.g: [apple,sausage,banana,cheese]

Response:
Try this:
```
dataframe[column].value_counts().index.tolist()
['apple', 'sausage', 'banana', 'cheese']
```


Question:
Pandas: convert dtype 'object' to int I've read an SQL query into Pandas and the values are coming in as dtype 'object', although they are strings, dates and integers. I am able to convert the date 'object' to a Pandas datetime dtype, but I'm getting an error when trying to convert the string and integers. Here is an example:
```

```python
import pandas as pd
```

```python
df = pd.read_sql_query('select * from my_table', conn)
```

```python
df
#Output
#    id    date          purchase
# 1  abc1  2016-05-22    1
# 2  abc2  2016-05-29    0
# 3  abc3  2016-05-22    2
# 4  abc4  2016-05-22    0
```

```python
df.dtypes
#Output
# id          object
# date        object
# purchase    object
# dtype: object
#```
#Converting the df['date'] to a datetime works:
#```
```

```python
pd.to_datetime(df['date'])
#Output
# 1  2016-05-22
# 2  2016-05-29
# 3  2016-05-22
# 4  2016-05-22
# Name: date, dtype: datetime64[ns]
#```
#But I get an error when trying to convert the df['purchase'] to an integer:
#```
```

```python
df['purchase'].astype(int)
#Output
# ....
# pandas/lib.pyx in pandas.lib.astype_intsafe (pandas/lib.c:16667)()
# pandas/src/util.pxd in util.set_value_at (pandas/lib.c:67540)()
```

TypeError: long() argument must be a string or a number, not 'java.lang.Long'
```
NOTE: I get a similar error when I tried .astype('float') And when trying to convert to a string, nothing seems to happen.
```

```python
df['id'].apply(str)
#Output
# 1 abc1
# 2 abc2
# 3 abc3
# 4 abc4
# Name: id, dtype: object
#```
```

Response:
Documenting the answer that worked for me based on the comment by @piRSquared. I needed to convert to a string first, then an integer.
```

```python
df['purchase'].astype(str).astype(int)
#Output
#```
```


Question:
How to get the last N rows of a pandas DataFrame? I have pandas dataframe df1:
```none
STK_ID  RPT_Date  TClose   sales  discount
0   000568  20060331    3.69   5.975       NaN
1   000568  20060630    9.14  10.143       NaN
2   000568  20060930    9.49  13.854       NaN
3   000568  20061231   15.84  19.262       NaN
4   000568  20070331   17.00   6.803       NaN
5   000568  20070630   26.31  12.940       NaN
6   000568  20070930   39.12  19.977       NaN
7   000568  20071231   45.94  29.269       NaN
8   000568  20080331   38.75  12.668       NaN
9   000568  20080630   30.09  21.102       NaN
10  000568  20080930   26.00  30.769       NaN
```
I wanted to select the last 3 rows and tried df1.ix[-3:], but it returns all the rows. Why? How to get the last 3 rows of df1? I'm using pandas 0.10.1.

Response: Don't forget DataFrame.tail! e.g. df1.tail(10)


Question:
pandas equivalent of R's cbind (concatenate/stack vectors horizontally) Suppose I have two dataframes:
```py
import pandas as pd

test1 = pd.DataFrame([1,2,3,4,5])

test2 = pd.DataFrame([4,2,1,3,7])
```
I tried test1.append(test2) but it is the equivalent of R's rbind. How can I combine the two as two columns of a dataframe similar to the cbind function in R?

Response:
```
test3 = pd.concat([test1, test2], axis=1)
test3.columns = ['a','b']
```
(But see the detailed answer by @feng-mai, below)


Question:
Keep certain columns in a pandas DataFrame, deleting everything else Say I have a data table
```
1  2  3  4  5  6 ..  n
A   x  x  x  x  x  x ..  x
B   x  x  x  x  x  x ..  x
C   x  x  x  x  x  x ..  x
```
And I want to slim it down so that I only have, say, columns 3 and 5 deleting all other and maintaining the structure. How could I do this with pandas? I think I understand how to delete a single column, but I don't know how to save a select few and delete all others.

Response:
If you have a list of columns you can just select those:
```

```python
df
#Output
#1  2  3  4  5  6
#A  x  x  x  x  x  x
#B  x  x  x  x  x  x
#C  x  x  x  x  x  x
```

```python
col_list = [3, 5]
```

```python
df = df[col_list]
```

```python
df
#Output
#3  5
#A  x  x
#B  x  x
#C  x  x
#```
```


Question: How to draw vertical lines on a given plot Given a plot of a signal in time representation, how can I draw lines marking the corresponding time index? Specifically, given a signal plot with a time index ranging from 0 to 2.6 (seconds), I want to draw vertical red lines indicating the corresponding time index for the list [0.22058956, 0.33088437, 2.20589566]. How can I do it?

Response:
The standard way to add vertical lines that will cover your entire plot window without you having to specify their actual height is plt.axvline
```
import matplotlib.pyplot as plt

plt.axvline(x=0.22058956)
plt.axvline(x=0.33088437)
plt.axvline(x=2.20589566)
```
OR
```
xcoords = [0.22058956, 0.33088437, 2.20589566]
for xc in xcoords:
    plt.axvline(x=xc)
```
You can use many of the keywords available for other plot commands (e.g. color, linestyle, linewidth ...). You can pass in keyword arguments ymin and ymax if you like in axes corrdinates (e.g. ymin=0.25, ymax=0.75 will cover the middle half of the plot). There are corresponding functions for horizontal lines (axhline) and rectangles (axvspan).


Question: Drop columns whose name contains a specific string from pandas DataFrame I have a pandas dataframe with the following column names: Result1, Test1, Result2, Test2, Result3, Test3, etc... I want to drop all the columns whose name contains the word "Test". The numbers of such columns is not static but depends on a previous function. How can I do that?

Response:
Here is one way to do this:
```
df = df[df.columns.drop(list(df.filter(regex='Test')))]
```


Question:
What is the difference between pandas agg and apply function? I can't figure out the difference between Pandas .aggregate and .apply functions. Take the following as an example: I load a dataset, do a groupby, define a simple function, and either user .agg or .apply. As you may see, the printing statement within my function results in the same output after using .agg and .apply. The result, on the other hand is different. Why is that?
```
import pandas
import pandas as pd
iris = pd.read_csv('iris.csv')
by_species = iris.groupby('Species')
def f(x):
    ...:     print type(x)
    ...:     print x.head(3)
    ...:     return 1
```
Using apply:
```
by_species.apply(f)
#<class 'pandas.core.frame.DataFrame'>
#   Sepal.Length  Sepal.Width  Petal.Length  Petal.Width Species
#0           5.1          3.5           1.4          0.2  setosa
#1           4.9          3.0           1.4          0.2  setosa
#2           4.7          3.2           1.3          0.2  setosa
#<class 'pandas.core.frame.DataFrame'>
#   Sepal.Length  Sepal.Width  Petal.Length  Petal.Width Species
#0           5.1          3.5           1.4          0.2  setosa
#1           4.9          3.0           1.4          0.2  setosa
#2           4.7          3.2           1.3          0.2  setosa
#<class 'pandas.core.frame.DataFrame'>
#    Sepal.Length  Sepal.Width  Petal.Length  Petal.Width     Species
#50           7.0          3.2           4.7          1.4  versicolor
#51           6.4          3.2           4.5          1.5  versicolor
#52           6.9          3.1           4.9          1.5  versicolor
#<class 'pandas.core.frame.DataFrame'>
#     Sepal.Length  Sepal.Width  Petal.Length  Petal.Width    Species
#100           6.3          3.3           6.0          2.5  virginica
#101           5.8          2.7           5.1          1.9  virginica
#102           7.1          3.0           5.9          2.1  virginica
#Out[33]:
#Species
#setosa        1
#versicolor    1
#virginica     1
#dtype: int64
```
Using agg
```
by_species.agg(f)
#<class 'pandas.core.frame.DataFrame'>
#   Sepal.Length  Sepal.Width  Petal.Length  Petal.Width Species
#0           5.1          3.5           1.4          0.2  setosa
#1           4.9          3.0           1.4          0.2  setosa
#2           4.7          3.2           1.3          0.2  setosa
#<class 'pandas.core.frame.DataFrame'>
#    Sepal.Length  Sepal.Width  Petal.Length  Petal.Width     Species
#50           7.0          3.2           4.7          1.4  versicolor
#51           6.4          3.2           4.5          1.5  versicolor
#52           6.9          3.1           4.9          1.5  versicolor
#<class 'pandas.core.frame.DataFrame'>
#     Sepal.Length  Sepal.Width  Petal.Length  Petal.Width    Species
#100           6.3          3.3           6.0          2.5  virginica
#101           5.8          2.7           5.1          1.9  virginica
#102           7.1          3.0           5.9          2.1  virginica
#Out[34]:
#           Sepal.Length  Sepal.Width  Petal.Length  Petal.Width
#Species
#setosa                 1            1             1            1
#versicolor             1            1             1            1
#virginica              1            1             1            1
```

Response: apply applies the function to each group (your Species). Your function returns 1, so you end up with 1 value for each of 3 groups. agg aggregates each column (feature) for each group, so you end up with one value per column per group. Do read the groupby docs, they're quite helpful. There are also a bunch of tutorials floating around the web.


Question:
Convert commas decimal separators to dots within a Dataframe I am importing a CSV file like the one below, using pandas.read_csv:
```
df = pd.read_csv(Input, delimiter=";")
```
Example of CSV file:
```
10;01.02.2015 16:58;01.02.2015 16:58;-0.59;0.1;-4.39;NotApplicable;0.79;0.2
11;01.02.2015 16:58;01.02.2015 16:58;-0.57;0.2;-2.87;NotApplicable;0.79;0.21
```
The problem is that when I later on in my code try to use these values I get this error: TypeError: can't multiply sequence by non-int of type 'float' The error is because the number I'm trying to use is not written with a dot (.) as a decimal separator but a comma(,). After manually changing the commas to a dots my program works. I can't change the format of my input, and thus have to replace the commas in my DataFrame in order for my code to work, and I want python to do this without the need of doing it manually. Do you have any suggestions?

Response:
pandas.read_csv has a decimal parameter for this. I.e. try with:
```
df = pd.read_csv(Input, delimiter=";", decimal=",")
```


Question:
Add months to a date in Pandas I'm trying to figure out how to add 3 months to a date in a Pandas dataframe, while keeping it in the date format, so I can use it to lookup a range. This is what I've tried:
```
#create dataframe
df = pd.DataFrame([pd.Timestamp('20161011'),
                   pd.Timestamp('20161101') ], columns=['date'])

#create a future month period
plus_month_period = 3

#calculate date + future period
df['future_date'] = plus_month_period.astype("timedelta64[M]")
```
However, I get the following error:
```
AttributeError: 'int' object has no attribute 'astype'
```

Response:
You could use pd.DateOffset
```

```python
df.date + pd.DateOffset(months=plus_month_period)
#Output
#0   2017-01-11
#1   2017-02-01
#Name: date, dtype: datetime64[ns]
#```
#Details
#```
```

```python
df
#Output
#date
#0 2016-10-11
#1 2016-11-01
```

```python
plus_month_period
#Output
#3
#```
```


Question:
Deleting multiple columns based on column names I have some data and when I import it, I get the following unneeded columns. I'm looking for an easy way to delete all of these.
```
'Unnamed: 24', 'Unnamed: 25', 'Unnamed: 26', 'Unnamed: 27',
'Unnamed: 28', 'Unnamed: 29', 'Unnamed: 30', 'Unnamed: 31',
'Unnamed: 32', 'Unnamed: 33', 'Unnamed: 34', 'Unnamed: 35',
'Unnamed: 36', 'Unnamed: 37', 'Unnamed: 38', 'Unnamed: 39',
'Unnamed: 40', 'Unnamed: 41', 'Unnamed: 42', 'Unnamed: 43',
'Unnamed: 44', 'Unnamed: 45', 'Unnamed: 46', 'Unnamed: 47',
'Unnamed: 48', 'Unnamed: 49', 'Unnamed: 50', 'Unnamed: 51',
'Unnamed: 52', 'Unnamed: 53', 'Unnamed: 54', 'Unnamed: 55',
'Unnamed: 56', 'Unnamed: 57', 'Unnamed: 58', 'Unnamed: 59',
'Unnamed: 60'
```
They are indexed by 0-indexing so I tried something like
```
df.drop(df.columns[[22, 23, 24, 25,
26, 27, 28, 29, 30, 31, 32 ,55]], axis=1, inplace=True)
```
But this isn't very efficient. I tried writing some for loops but this struck me as bad Pandas behaviour. Hence i ask the question here. I've seen some examples which are similar (Drop multiple columns in pandas) but this doesn't answer my question.

Response:
By far the simplest approach is:
```
yourdf.drop(['columnheading1', 'columnheading2'], axis=1, inplace=True)
```


Question:
How to plot one single data point? I have the following code to plot a line and a point:
```
df = pd.DataFrame({'x': [1, 2, 3], 'y': [3, 4, 6]})
point = pd.DataFrame({'x': [2], 'y': [5]})
ax = df.plot(x='x', y='y', label='line')
ax = point.plot(x='x', y='y', ax=ax, style='r-', label='point')
```
How do I get the single data point to show up?

Response:
To plot a single point you can do something like this:
```
plt.plot([x], [y], marker='o', markersize=3, color="red")
```


Question:
Normalizing pandas DataFrame rows by their sums What is the most idiomatic way to normalize each row of a pandas DataFrame? Normalizing the columns is easy, so one (very ugly!) option is:
```
(df.T / df.T.sum()).T
```
Pandas broadcasting rules prevent df / df.sum(axis=1) from doing this

Response:
To overcome the broadcasting issue, you can use the div method:
```
df.div(df.sum(axis=1), axis=0)
```
See pandas User Guide: Matching / broadcasting behavior


Question:
Pandas DataFrame to List of Lists It's easy to turn a list of lists into a pandas dataframe:
```
import pandas as pd
df = pd.DataFrame([[1,2,3],[3,4,5]])
```
But how do I turn df back into a list of lists?
```
lol = df.what_to_do_now?
print lol
# [[1,2,3],[3,4,5]]
```

Response:
You could access the underlying array and call its tolist method:
```

```python
df = pd.DataFrame([[1,2,3],[3,4,5]])
```

```python
lol = df.values.tolist()
```

```python
lol
#Output
#[[1L, 2L, 3L], [3L, 4L, 5L]]
#```
```