Question: pandas resample documentation So I completely understand how to use resample, but the documentation does not do a good job explaining the options. So most options in the resample function are pretty straight forward except for these two: rule : the offset string or object representing target conversion how : string, method for down- or re-sampling, default to ‘mean’ So from looking at as many examples as I found online I can see for rule you can do 'D' for day, 'xMin' for minutes, 'xL' for milliseconds, but that is all I could find. for how I have seen the following: 'first', np.max, 'last', 'mean', and 'n1n2n3n4...nx' where nx is the first letter of each column index. So is there somewhere in the documentation that I am missing that displays every option for pandas.resample's rule and how inputs? If yes, where because I could not find it. If no, what are all the options for them?

Response:
```
B         business day frequency
C         custom business day frequency (experimental)
D         calendar day frequency
W         weekly frequency
M         month end frequency
SM        semi-month end frequency (15th and end of month)
BM        business month end frequency
CBM       custom business month end frequency
MS        month start frequency
SMS       semi-month start frequency (1st and 15th)
BMS       business month start frequency
CBMS      custom business month start frequency
Q         quarter end frequency
BQ        business quarter endfrequency
QS        quarter start frequency
BQS       business quarter start frequency
A         year end frequency
BA, BY    business year end frequency
AS, YS    year start frequency
BAS, BYS  business year start frequency
BH        business hour frequency
H         hourly frequency
T, min    minutely frequency
S         secondly frequency
L, ms     milliseconds
U, us     microseconds
N         nanoseconds
```
See the timeseries documentation. It includes a list of offsets (and 'anchored' offsets), and a section about resampling. Note that there isn't a list of all the different how options, because it can be any NumPy array function and any function that is available via groupby dispatching can be passed to how by name.


Question: pandas how to check dtype for all columns in a dataframe? It seems that dtype only work for pandas.DataFrame.Series, right? Is there a function to display data types of all columns at once?

Response:
The singular form dtype is used to check the data type for a single column. And the plural form dtypes is for data frame which returns data types for all columns. Essentially: For a single column:
```
dataframe.column.dtype
```
For all columns:
```
dataframe.dtypes
```
Example:
```
import pandas as pd
df = pd.DataFrame({'A': [1,2,3], 'B': [True, False, False], 'C': ['a', 'b', 'c']})

df.A.dtype
# dtype('int64')
df.B.dtype
# dtype('bool')
df.C.dtype
# dtype('O')

df.dtypes
#A     int64
#B      bool
#C    object
#dtype: object
```


Question:
Pandas join issue: columns overlap but no suffix specified I have the following data frames:
```
print(df_a)
     mukey  DI  PI
0   100000  35  14
1  1000005  44  14
2  1000006  44  14
3  1000007  43  13
4  1000008  43  13

print(df_b)
    mukey  niccdcd
0  190236        4
1  190237        6
2  190238        7
3  190239        4
4  190240        7
```
When I try to join these data frames:
```
join_df = df_a.join(df_b, on='mukey', how='left')
```
I get the error:
```
*** ValueError: columns overlap but no suffix specified: Index([u'mukey'], dtype='object')
```
Why is this so? The data frames do have common 'mukey' values.

Response:
Your error on the snippet of data you posted is a little cryptic, in that because there are no common values, the join operation fails because the values don't overlap it requires you to supply a suffix for the left and right hand side:
```

```python
df_a.join(df_b, on='mukey', how='left', lsuffix='_left', rsuffix='_right')
#Output
#mukey_left  DI  PI  mukey_right  niccdcd
#index
#0          100000  35  14          NaN      NaN
#1         1000005  44  14          NaN      NaN
#2         1000006  44  14          NaN      NaN
#3         1000007  43  13          NaN      NaN
#4         1000008  43  13          NaN      NaN
#```
#merge works because it doesn't have this restriction:
#```
```

```python
df_a.merge(df_b, on='mukey', how='left')
#Output
#mukey  DI  PI  niccdcd
#0   100000  35  14      NaN
#1  1000005  44  14      NaN
#2  1000006  44  14      NaN
#3  1000007  43  13      NaN
#4  1000008  43  13      NaN
#```
```


Question:
Get total of Pandas column I have a Pandas data frame, as shown below, with multiple columns and would like to get the total of column, MyColumn.
```none
X           MyColumn      Y              Z
0          A           84         13.0           69.0
1          B           76         77.0          127.0
2          C           28         69.0           16.0
3          D           28         28.0           31.0
4          E           19         20.0           85.0
5          F           84        193.0           70.0
```
Expected Output I'd have expected the output to be the total of this column: 319. Or alternatively, I would like df to be edited with a new row entitled TOTAL containing the total:
```none
X           MyColumn      Y              Z
0          A           84         13.0           69.0
1          B           76         77.0          127.0
2          C           28         69.0           16.0
3          D           28         28.0           31.0
4          E           19         20.0           85.0
5          F           84        193.0           70.0
TOTAL                  319
```
I have attempted to get the sum of the column using groupby and .sum():
```py
Total = df.groupby['MyColumn'].sum()
```
This causes the following error:
```none
TypeError: 'instancemethod' object has no attribute '__getitem__'
```

Response:
You should use sum:
```
Total = df['MyColumn'].sum()
print(Total)
319
```
Then you use loc with Series, in that case the index should be set as the same as the specific column you need to sum:
```
df.loc['Total'] = pd.Series(df['MyColumn'].sum(), index=['MyColumn'])
print(df)
         X  MyColumn      Y      Z
0        A      84.0   13.0   69.0
1        B      76.0   77.0  127.0
2        C      28.0   69.0   16.0
3        D      28.0   28.0   31.0
4        E      19.0   20.0   85.0
5        F      84.0  193.0   70.0
Total  NaN     319.0    NaN    NaN
```
because if you pass scalar, the values of all rows will be filled:
```
df.loc['Total'] = df['MyColumn'].sum()
print(df)
         X  MyColumn      Y      Z
0        A        84   13.0   69.0
1        B        76   77.0  127.0
2        C        28   69.0   16.0
3        D        28   28.0   31.0
4        E        19   20.0   85.0
5        F        84  193.0   70.0
Total  319       319  319.0  319.0
```
Two other solutions are with at, and ix see the applications below:
```
df.at['Total', 'MyColumn'] = df['MyColumn'].sum()
print(df)
         X  MyColumn      Y      Z
0        A      84.0   13.0   69.0
1        B      76.0   77.0  127.0
2        C      28.0   69.0   16.0
3        D      28.0   28.0   31.0
4        E      19.0   20.0   85.0
5        F      84.0  193.0   70.0
Total  NaN     319.0    NaN    NaN
```
```
df.ix['Total', 'MyColumn'] = df['MyColumn'].sum()
print(df)
         X  MyColumn      Y      Z
0        A      84.0   13.0   69.0
1        B      76.0   77.0  127.0
2        C      28.0   69.0   16.0
3        D      28.0   28.0   31.0
4        E      19.0   20.0   85.0
5        F      84.0  193.0   70.0
Total  NaN     319.0    NaN    NaN
```
Note: Since Pandas v0.20, ix has been deprecated. Use loc or iloc instead.


Question: Python Pandas merge only certain columns Is it possible to only merge some columns? I have a DataFrame df1 with columns x, y, z, and df2 with columns x, a ,b, c, d, e, f, etc. I want to merge the two DataFrames on x, but I only want to merge columns df2.a, df2.b - not the entire DataFrame. The result would be a DataFrame with x, y, z, a, b. I could merge then delete the unwanted columns, but it seems like there is a better method.

Response:
You want to use TWO brackets, so if you are doing a VLOOKUP sort of action:
```
df = pd.merge(df,df2[['Key_Column','Target_Column']],on='Key_Column', how='left')
```
This will give you everything in the original df + add that one corresponding column in df2 that you want to join.


Question:
How to read a .xlsx file using the pandas Library in iPython? I want to read a .xlsx file using the Pandas Library of python and port the data to a postgreSQL table. All I could do up until now is:
```
import pandas as pd
data = pd.ExcelFile("*File Name*")
```
Now I know that the step got executed successfully, but I want to know how i can parse the excel file that has been read so that I can understand how the data in the excel maps to the data in the variable data. I learnt that data is a Dataframe object if I'm not wrong. So How do i parse this dataframe object to extract each line row by row.

Response:
I usually create a dictionary containing a DataFrame for every sheet:
```
xl_file = pd.ExcelFile(file_name)

dfs = {sheet_name: xl_file.parse(sheet_name)
          for sheet_name in xl_file.sheet_names}
```
Update: In pandas version 0.21.0+ you will get this behavior more cleanly by passing sheet_name=None to read_excel:
```
dfs = pd.read_excel(file_name, sheet_name=None)
```
In 0.20 and prior, this was sheetname rather than sheet_name (this is now deprecated in favor of the above):
```
dfs = pd.read_excel(file_name, sheetname=None)
```


Question:
Get list of pandas dataframe columns based on data type If I have a dataframe with the following columns:
```
1. NAME                                     object
2. On_Time                                      object
3. On_Budget                                    object
4. %actual_hr                                  float64
5. Baseline Start Date                  datetime64[ns]
6. Forecast Start Date                  datetime64[ns]
```
I would like to be able to say: for this dataframe, give me a list of the columns which are of type 'object' or of type 'datetime'? I have a function which converts numbers ('float64') to two decimal places, and I would like to use this list of dataframe columns, of a particular type, and run it through this function to convert them all to 2dp. Maybe something like:
```
For c in col_list: if c.dtype = "Something"
list[]
List.append(c)?
```

Response:
If you want a list of columns of a certain type, you can use groupby:
```

```python
df = pd.DataFrame([[1, 2.3456, 'c', 'd', 78]], columns=list("ABCDE"))
```

```python
df
#Output
#   A       B  C  D   E
#0  1  2.3456  c  d  78
```

[1 rows x 5 columns]

```python
df.dtypes
#Output
#A      int64
#B    float64
#C     object
#D     object
#E      int64
#dtype: object
```

```python
g = df.columns.to_series().groupby(df.dtypes).groups
```

```python
g
#Output
#{dtype('int64'): ['A', 'E'], dtype('float64'): ['B'], dtype('O'): ['C', 'D']}
```

```python
{k.name: v for k, v in g.items()}
#Output
#{'object': ['C', 'D'], 'int64': ['A', 'E'], 'float64': ['B']}
#```
```


Question:
Replacing column values in a pandas DataFrame I'm trying to replace the values in one column of a dataframe. The column ('female') only contains the values 'female' and 'male'. I have tried the following:
```
w['female']['female']='1'
w['female']['male']='0'
```
But receive the exact same copy of the previous results. I would ideally like to get some output which resembles the following loop element-wise.
```
if w['female'] =='female':
    w['female'] = '1';
else:
    w['female'] = '0';
```
I've looked through the gotchas documentation (http://pandas.pydata.org/pandas-docs/stable/gotchas.html) but cannot figure out why nothing happens. Any help will be appreciated.

Response:
If I understand right, you want something like this:
```
w['female'] = w['female'].map({'female': 1, 'male': 0})
```
(Here I convert the values to numbers instead of strings containing numbers. You can convert them to "1" and "0", if you really want, but I'm not sure why you'd want that.) The reason your code doesn't work is because using ['female'] on a column (the second 'female' in your w['female']['female']) doesn't mean "select rows where the value is 'female'". It means to select rows where the index is 'female', of which there may not be any in your DataFrame.


Question:
Print very long string in Pandas dataframe I have a Pandas data frame containing a very long string:
```py
df = pd.DataFrame({'one' : ['one', 'two',
      'This is very long string very long string very long string veryvery long string']})
```
When I print it, I see only part of the string. I tried: using print(df.iloc[2]). using to_html. using to_string. to increase column width by using Pandas display option. I also do not get how set_printoptions would solve this.

Response:
You can use options.display.max_colwidth to specify you want to see more in the default representation:
```

```python
df
#Output
#one
#0                                                one
#1                                                two
#2  This is very long string very long string very...
```

```python
pd.options.display.max_colwidth
#Output
#50
```

```python
pd.options.display.max_colwidth = 100
```

```python
df
#Output
#one
#0                                                                              one
#1                                                                              two
#2  This is very long string very long string very long string veryvery long string
#```
#And indeed, if you just want to inspect the one value, by accessing it (as a scalar, not as a row as df.iloc[2] does) you also see the full string:
#```
```

```python
df.iloc[2,0]    # or df.loc[2,'one']
#Output
#'This is very long string very long string very long string veryvery long string'
#```
```


Question:
Convert unix time to readable date in pandas dataframe I have a dataframe with unix times and prices in it. I want to convert the index column so that it shows in human readable dates. So for instance I have date as 1349633705 in the index column but I'd want it to show as 10/07/2012 (or at least 10/07/2012 18:15). For some context, here is the code I'm working with and what I've tried already:
```
import json
import urllib2
from datetime import datetime
response = urllib2.urlopen('http://blockchain.info/charts/market-price?&format=json')
data = json.load(response)
df = DataFrame(data['values'])
df.columns = ["date","price"]
#convert dates
df.date = df.date.apply(lambda d: datetime.strptime(d, "%Y-%m-%d"))
df.index = df.date
```
As you can see I'm using df.date = df.date.apply(lambda d: datetime.strptime(d, "%Y-%m-%d")) here which doesn't work since I'm working with integers, not strings. I think I need to use datetime.date.fromtimestamp but I'm not quite sure how to apply this to the whole of df.date. Thanks.

Response:
These appear to be seconds since epoch.
```

```python
df = DataFrame(data['values'])
```

```python
df.columns = ["date","price"]
```

```python
df
#Output
#<class 'pandas.core.frame.DataFrame'>
#Int64Index: 358 entries, 0 to 357
#Data columns (total 2 columns):
#date     358  non-null values
#price    358  non-null values
#dtypes: float64(1), int64(1)
```

```python
df.head()
#Output
#date  price
#0  1349720105  12.08
#1  1349806505  12.35
#2  1349892905  12.15
#3  1349979305  12.19
#4  1350065705  12.15
```

```python
df['date'] = pd.to_datetime(df['date'],unit='s')
```

```python
df.head()
#Output
#date  price
#0 2012-10-08 18:15:05  12.08
#1 2012-10-09 18:15:05  12.35
#2 2012-10-10 18:15:05  12.15
#3 2012-10-11 18:15:05  12.19
#4 2012-10-12 18:15:05  12.15
```

```python
df.dtypes
#Output
#date     datetime64[ns]
#price           float64
#dtype: object
#```
```


Question:
Naming returned columns in Pandas aggregate function? [duplicate] This question already has answers here: Multiple aggregations of the same column using pandas GroupBy.agg() (5 answers) Closed 6 years ago. I'm having trouble with Pandas' groupby functionality. I've read the documentation, but I can't see to figure out how to apply aggregate functions to multiple columns and have custom names for those columns. This comes very close, but the data structure returned has nested column headings:
```
data.groupby("Country").agg({
  "column1": {"foo": sum()},
  "column2": {"mean": np.mean, "std": np.std}
})
```
(ie. I want to take the mean and std of column2, but return those columns as "mean" and "std") What am I missing?

Response:
For pandas >= 0.25 The functionality to name returned aggregate columns has been reintroduced in the master branch and is targeted for pandas 0.25. The new syntax is .agg(new_col_name=('col_name', 'agg_func'). Detailed example from the PR linked above:
```

```python
df = pd.DataFrame({'kind': ['cat', 'dog', 'cat', 'dog'],
   ...:                    'height': [9.1, 6.0, 9.5, 34.0],
   ...:                    'weight': [7.9, 7.5, 9.9, 198.0]})
   ...:
```

```python
df
#Output
#kind  height  weight
#0  cat     9.1     7.9
#1  dog     6.0     7.5
#2  cat     9.5     9.9
#3  dog    34.0   198.0
```

```python
df.groupby('kind').agg(min_height=('height', 'min'),
                               max_weight=('weight', 'max'))
#Output
#min_height  max_weight
#kind
#cat          9.1         9.9
#dog          6.0       198.0
#```
#It will also be possible to use multiple lambda expressions with this syntax and the two-step rename syntax I suggested earlier (below) as per this PR. Again, copying from the example in the PR:
#```
```

```python
df = pd.DataFrame({"A": ['a', 'a'], 'B': [1, 2], 'C': [3, 4]})
```

```python
df.groupby("A").agg({'B': [lambda x: 0, lambda x: 1]})
#Output
#B
#  <lambda> <lambda 1>
#A
#a        0          1
#```
#and then .rename(), or in one go:
#```
```

```python
df.groupby("A").agg(b=('B', lambda x: 0), c=('B', lambda x: 1))
#Output
#b  c
#A
#a  0  0
#```
#For pandas < 0.25 The currently accepted answer by unutbu describes are great way of doing this in pandas versions <= 0.20. However, as of pandas 0.20, using this method raises a warning indicating that the syntax will not be available in future versions of pandas. Series: FutureWarning: using a dict on a Series for aggregation is deprecated and will be removed in a future version DataFrames: FutureWarning: using a dict with renaming is deprecated and will be removed in a future version According to the pandas 0.20 changelog, the recommended way of renaming columns while aggregating is as follows.
#```
## Create a sample data frame
#df = pd.DataFrame({'A': [1, 1, 1, 2, 2],
#                   'B': range(5),
#                   'C': range(5)})
#
## ==== SINGLE COLUMN (SERIES) ====
## Syntax soon to be deprecated
#df.groupby('A').B.agg({'foo': 'count'})
## Recommended replacement syntax
#df.groupby('A').B.agg(['count']).rename(columns={'count': 'foo'})
#
## ==== MULTI COLUMN ====
## Syntax soon to be deprecated
#df.groupby('A').agg({'B': {'foo': 'sum'}, 'C': {'bar': 'min'}})
## Recommended replacement syntax
#df.groupby('A').agg({'B': 'sum', 'C': 'min'}).rename(columns={'B': 'foo', 'C': 'bar'})
## As the recommended syntax is more verbose, parentheses can
## be used to introduce line breaks and increase readability
#(df.groupby('A')
#    .agg({'B': 'sum', 'C': 'min'})
#    .rename(columns={'B': 'foo', 'C': 'bar'})
#)
#```
#Please see the 0.20 changelog for additional details. Update 2017-01-03 in response to @JunkMechanic's comment. With the old style dictionary syntax, it was possible to pass multiple lambda functions to .agg, since these would be renamed with the key in the passed dictionary:
#```
#>>> df.groupby('A').agg({'B': {'min': lambda x: x.min(), 'max': lambda x: x.max()}})
#
#    B
#  max min
#A
#1   2   0
#2   4   3
#```
#Multiple functions can also be passed to a single column as a list:
#```
#>>> df.groupby('A').agg({'B': [np.min, np.max]})
#
#     B
#  amin amax
#A
#1    0    2
#2    3    4
#```
#However, this does not work with lambda functions, since they are anonymous and all return , which causes a name collision:
#```
#>>> df.groupby('A').agg({'B': [lambda x: x.min(), lambda x: x.max]})
#SpecificationError: Function names must be unique, found multiple named <lambda>
#```
#To avoid the SpecificationError, named functions can be defined a priori instead of using lambda. Suitable function names also avoid calling .rename on the data frame afterwards. These functions can be passed with the same list syntax as above:
#```
#>>> def my_min(x):
#>>>     return x.min()
#
#>>> def my_max(x):
#>>>     return x.max()
#
#>>> df.groupby('A').agg({'B': [my_min, my_max]})
#
#       B
#  my_min my_max
#A
#1      0      2
#2      3      4
#```
```


Question:
Pandas: sum DataFrame rows for given columns I have the following DataFrame:
```

```python
df = pd.DataFrame({'a': [1, 2, 3],
                   'b': [2, 3, 4],
                   'c': ['dd', 'ee', 'ff'],
                   'd': [5, 9, 1]})

df
#Output
#a  b   c  d
#0  1  2  dd  5
#1  2  3  ee  9
#2  3  4  ff  1
#```
#I would like to add a column 'e' which is the sum of columns 'a', 'b' and 'd'. Going across forums, I thought something like this would work:
#```
#df['e'] = df[['a', 'b', 'd']].map(sum)
#```
#But it didn't. I would like to know the appropriate operation with the list of columns ['a', 'b', 'd'] and df as inputs.
```

Response:
You can just sum and set axis=1 to sum the rows, which will ignore non-numeric columns; from pandas 2.0+ you also need to specify numeric_only=True.
```

```python
df = pd.DataFrame({'a': [1,2,3], 'b': [2,3,4], 'c':['dd','ee','ff'], 'd':[5,9,1]})
df['e'] = df.sum(axis=1, numeric_only=True)
df
#Output
#a  b   c  d   e
#0  1  2  dd  5   8
#1  2  3  ee  9  14
#2  3  4  ff  1   8
#```
#If you want to just sum specific columns then you can create a list of the columns and remove the ones you are not interested in:
#```
```

```python
col_list= list(df)
col_list.remove('d')
col_list
#Output
#['a', 'b', 'c']
```

```python
df['e'] = df[col_list].sum(axis=1)
df
#Output
#a  b   c  d  e
#0  1  2  dd  5  3
#1  2  3  ee  9  5
#2  3  4  ff  1  7
#```
#sum docs
```


Question:
How to access subdataframes of pandas groupby by key How do I access the corresponding groupby dataframe in a groupby object by the key? With the following groupby:
```
rand = np.random.RandomState(1)
df = pd.DataFrame({'A': ['foo', 'bar'] * 3,
                   'B': rand.randn(6),
                   'C': rand.randint(0, 20, 6)})
gb = df.groupby(['A'])
```
I can iterate through it to get the keys and groups:
```

```python
for k, gp in gb:
             print 'key=' + str(k)
             print gp
key=bar
     A         B   C
1  bar -0.611756  18
3  bar -1.072969  10
5  bar -2.301539  18
key=foo
     A         B   C
0  foo  1.624345   5
2  foo -0.528172  11
4  foo  0.865408  14
```
I would like to be able to access a group by its key:
```
```

```python
gb['foo']
#Output
#A         B   C
#0  foo  1.624345   5
#2  foo -0.528172  11
#4  foo  0.865408  14
#```
#But when I try doing that with gb[('foo',)] I get this weird pandas.core.groupby.DataFrameGroupBy object thing which doesn't seem to have any methods that correspond to the DataFrame I want. The best I could think of is:
#```
```

```python
def gb_df_key(gb, key, orig_df):
             ix = gb.indices[key]
             return orig_df.ix[ix]

         gb_df_key(gb, 'foo', df)
#Output
#A         B   C
#0  foo  1.624345   5
#2  foo -0.528172  11
#4  foo  0.865408  14
#```
#but this is kind of nasty, considering how nice pandas usually is at these things. What's the built-in way of doing this?
```

Response:
You can use the get_group method:
```

```python
gb.get_group('foo')
#Output
#A         B   C
#0  foo  1.624345   5
#2  foo -0.528172  11
#4  foo  0.865408  14
#```
#Note: This doesn't require creating an intermediary dictionary / copy of every subdataframe for every group, so will be much more memory-efficient than creating the naive dictionary with dict(iter(gb)). This is because it uses data-structures already available in the groupby object. You can select different columns using the groupby slicing:
#```
```

```python
gb[["A", "B"]].get_group("foo")
#Output
#A         B
#0  foo  1.624345
#2  foo -0.528172
#4  foo  0.865408
```

```python
gb["C"].get_group("foo")
#Output
#0     5
#2    11
#4    14
#Name: C, dtype: int64
#```
```


Question:
pandas unique values multiple columns
```
df = pd.DataFrame({'Col1': ['Bob', 'Joe', 'Bill', 'Mary', 'Joe'],
                   'Col2': ['Joe', 'Steve', 'Bob', 'Bob', 'Steve'],
                   'Col3': np.random.random(5)})
```
What is the best way to return the unique values of 'Col1' and 'Col2'? The desired output is
```
'Bob', 'Joe', 'Bill', 'Mary', 'Steve'
```

Response:
pd.unique returns the unique values from an input array, or DataFrame column or index. The input to this function needs to be one-dimensional, so multiple columns will need to be combined. The simplest way is to select the columns you want and then view the values in a flattened NumPy array. The whole operation looks like this:
```

```python
pd.unique(df[['Col1', 'Col2']].values.ravel('K'))
#Output
#array(['Bob', 'Joe', 'Bill', 'Mary', 'Steve'], dtype=object)
#```
#Note that ravel() is an array method that returns a view (if possible) of a multidimensional array. The argument 'K' tells the method to flatten the array in the order the elements are stored in the memory (pandas typically stores underlying arrays in Fortran-contiguous order; columns before rows). This can be significantly faster than using the method's default 'C' order. An alternative way is to select the columns and pass them to np.unique:
#```
```

```python
np.unique(df[['Col1', 'Col2']].values)
#Output
#array(['Bill', 'Bob', 'Joe', 'Mary', 'Steve'], dtype=object)
#```
#There is no need to use ravel() here as the method handles multidimensional arrays. Even so, this is likely to be slower than pd.unique as it uses a sort-based algorithm rather than a hashtable to identify unique values. The difference in speed is significant for larger DataFrames (especially if there are only a handful of unique values):
#```
```

```python
df1 = pd.concat([df]*100000, ignore_index=True) # DataFrame with 500000 rows
```

```python
%timeit np.unique(df1[['Col1', 'Col2']].values)
#Output
#1 loop, best of 3: 1.12 s per loop
```

```python
%timeit pd.unique(df1[['Col1', 'Col2']].values.ravel('K'))
#Output
#10 loops, best of 3: 38.9 ms per loop
```

```python
%timeit pd.unique(df1[['Col1', 'Col2']].values.ravel()) # ravel using C order
#Output
#10 loops, best of 3: 49.9 ms per loop
#```
```


Question: Python Pandas: How to read only first n rows of CSV files in? I have a very large data set and I can't afford to read the entire data set in. So, I'm thinking of reading only one chunk of it to train but I have no idea how to do it.

Response:
If you only want to read the first 999,999 (non-header) rows:
```
read_csv(..., nrows=999999)
```
If you only want to read rows 1,000,000 ... 1,999,999
```
read_csv(..., skiprows=1000000, nrows=999999)
```
nrows : int, default None Number of rows of file to read. Useful for reading pieces of large files* skiprows : list-like or integer Row numbers to skip (0-indexed) or number of rows to skip (int) at the start of the file and for large files, you'll probably also want to use chunksize: chunksize : int, default None Return TextFileReader object for iteration pandas.io.parsers.read_csv documentation


Question:
Find the column name which has the maximum value for each row I have a DataFrame like this one:
```none
Communications and Search   Business    General Lifestyle
0   0.745763    0.050847    0.118644    0.084746
0   0.333333    0.000000    0.583333    0.083333
0   0.617021    0.042553    0.297872    0.042553
0   0.435897    0.000000    0.410256    0.153846
0   0.358974    0.076923    0.410256    0.153846
```
I want to create a new column comprised of the column labels of each row’s maximum value. The desired output is like this:
```none
Communications and Search   Business    General Lifestyle  Max
0   0.745763    0.050847    0.118644    0.084746           Communications
0   0.333333    0.000000    0.583333    0.083333           Business
0   0.617021    0.042553    0.297872    0.042553           Communications
0   0.435897    0.000000    0.410256    0.153846           Communications
0   0.358974    0.076923    0.410256    0.153846           Business
```

Response:
You can use idxmax with axis=1 to find the column with the greatest value on each row:
```

```python
df.idxmax(axis=1)
#Output
#0    Communications
#1          Business
#2    Communications
#3    Communications
#4          Business
#dtype: object
#```
#To create the new column 'Max', use df['Max'] = df.idxmax(axis=1). To find the row index at which the maximum value occurs in each column, use df.idxmax() (or equivalently df.idxmax(axis=0)).
```


Question:
Removing index column in pandas when reading a csv I have the following code which imports a CSV file. There are 3 columns and I want to set the first two of them to variables. When I set the second column to the variable "efficiency" the index column is also tacked on. How can I get rid of the index column?
```
df = pd.DataFrame.from_csv('Efficiency_Data.csv', header=0, parse_dates=False)
energy = df.index
efficiency = df.Efficiency
print efficiency
```
I tried using
```
del df['index']
```
after I set
```
energy = df.index
```
which I found in another post but that results in "KeyError: 'index' "

Response:
When writing to and reading from a CSV file include the argument index=False and index_col=False, respectively. Follows an example: To write:
```
df.to_csv(filename, index=False)
```
and to read from the csv
```
df.read_csv(filename, index_col=False)
```
This should prevent the issue so you don't need to fix it later.


Question:
Conditional Replace Pandas [duplicate] This question already has answers here: Pandas - Case when & default in pandas (4 answers) Closed last year. I have a DataFrame, and I want to replace the values in a particular column that exceed a value with zero. I had thought this was a way of achieving this:
```
df[df.my_channel > 20000].my_channel = 0
```
If I copy the channel into a new data frame it's simple:
```
df2 = df.my_channel

df2[df2 > 20000] = 0
```
This does exactly what I want, but seems not to work with the channel as part of the original DataFrame.

Response:
.ix indexer works okay for pandas version prior to 0.20.0, but since pandas 0.20.0, the .ix indexer is deprecated, so you should avoid using it. Instead, you can use .loc or iloc indexers. You can solve this problem by:
```
mask = df.my_channel > 20000
column_name = 'my_channel'
df.loc[mask, column_name] = 0
```
Or, in one line,
```
df.loc[df.my_channel > 20000, 'my_channel'] = 0
```
mask helps you to select the rows in which df.my_channel > 20000 is True, while df.loc[mask, column_name] = 0 sets the value 0 to the selected rows where maskholds in the column which name is column_name. Update: In this case, you should use loc because if you use iloc, you will get a NotImplementedError telling you that iLocation based boolean indexing on an integer type is not available.


Question:
Combine Date and Time columns using pandas I have a pandas dataframe with the following columns:
```py
data = {'Date': ['01-06-2013', '02-06-2013', '02-06-2013', '02-06-2013', '02-06-2013', '03-06-2013', '03-06-2013', '03-06-2013', '03-06-2013', '04-06-2013'],
        'Time': ['23:00:00', '01:00:00', '21:00:00', '22:00:00', '23:00:00', '01:00:00', '21:00:00', '22:00:00', '23:00:00', '01:00:00']}
df = pd.DataFrame(data)

         Date      Time
0  01-06-2013  23:00:00
1  02-06-2013  01:00:00
2  02-06-2013  21:00:00
3  02-06-2013  22:00:00
4  02-06-2013  23:00:00
5  03-06-2013  01:00:00
6  03-06-2013  21:00:00
7  03-06-2013  22:00:00
8  03-06-2013  23:00:00
9  04-06-2013  01:00:00
```
How do I combine data['Date'] & data['Time'] to get the following? Is there a way of doing it using pd.to_datetime?
```
Date
01-06-2013 23:00:00
02-06-2013 01:00:00
02-06-2013 21:00:00
02-06-2013 22:00:00
02-06-2013 23:00:00
03-06-2013 01:00:00
03-06-2013 21:00:00
03-06-2013 22:00:00
03-06-2013 23:00:00
04-06-2013 01:00:00
```

Response:
It's worth mentioning that you may have been able to read this in directly e.g. if you were using read_csv using parse_dates=[['Date', 'Time']]. Assuming these are just strings you could simply add them together (with a space), allowing you to use to_datetime, which works without specifying the format= parameter
```

```python
df['Date'] + ' ' + df['Time']
#Output
#0    01-06-2013 23:00:00
#1    02-06-2013 01:00:00
#2    02-06-2013 21:00:00
#3    02-06-2013 22:00:00
#4    02-06-2013 23:00:00
#5    03-06-2013 01:00:00
#6    03-06-2013 21:00:00
#7    03-06-2013 22:00:00
#8    03-06-2013 23:00:00
#9    04-06-2013 01:00:00
#dtype: object
```

```python
pd.to_datetime(df['Date'] + ' ' + df['Time'])
#Output
#0   2013-01-06 23:00:00
#1   2013-02-06 01:00:00
#2   2013-02-06 21:00:00
#3   2013-02-06 22:00:00
#4   2013-02-06 23:00:00
#5   2013-03-06 01:00:00
#6   2013-03-06 21:00:00
#7   2013-03-06 22:00:00
#8   2013-03-06 23:00:00
#9   2013-04-06 01:00:00
#dtype: datetime64[ns]
#```
#Alternatively, without the + ' ', but the format= parameter must be used. Additionally, pandas is good at inferring the format to be converted to a datetime, however, specifying the exact format is faster.
#```py
#pd.to_datetime(df['Date'] + df['Time'], format='%m-%d-%Y%H:%M:%S')
#```
#Note: surprisingly (for me), this works fine with NaNs being converted to NaT, but it is worth worrying that the conversion (perhaps using the raise argument). %%timeit
#```py
## sample dataframe with 10000000 rows using df from the OP
#df = pd.concat([df for _ in range(1000000)]).reset_index(drop=True)
#
#%%timeit
#pd.to_datetime(df['Date'] + ' ' + df['Time'])
#[result]:
#1.73 s ± 10.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
#
#%%timeit
#pd.to_datetime(df['Date'] + df['Time'], format='%m-%d-%Y%H:%M:%S')
#[result]:
#1.33 s ± 9.88 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
#```
```


Question:
Modifying a subset of rows in a pandas dataframe Assume I have a pandas DataFrame with two columns, A and B. I'd like to modify this DataFrame (or create a copy) so that B is always NaN whenever A is 0. How would I achieve that? I tried the following
```
df['A'==0]['B'] = np.nan
```
and
```
df['A'==0]['B'].values.fill(np.nan)
```
without success.

Response:
Use .loc for label based indexing:
```
df.loc[df.A==0, 'B'] = np.nan
```
The df.A==0 expression creates a boolean series that indexes the rows, 'B' selects the column. You can also use this to transform a subset of a column, e.g.:
```
df.loc[df.A==0, 'B'] = df.loc[df.A==0, 'B'] / 2
```
I don't know enough about pandas internals to know exactly why that works, but the basic issue is that sometimes indexing into a DataFrame returns a copy of the result, and sometimes it returns a view on the original object. According to documentation here, this behavior depends on the underlying numpy behavior. I've found that accessing everything in one operation (rather than [one][two]) is more likely to work for setting.


Question: What are the differences between Pandas and NumPy+SciPy in Python? [closed] Closed. This question is opinion-based. It is not currently accepting answers. Want to improve this question? Because this question may lead to opinionated discussion, debate, and answers, it has been closed. You may edit the question if you feel you can improve it so that it requires answers that include facts and citations or a detailed explanation of the proposed solution. If edited, the question will be reviewed and might be reopened. Closed 10 years ago. Improve this question They both seem exceedingly similar and I'm curious as to which package would be more beneficial for financial data analysis.

Response: pandas provides high level data manipulation tools built on top of NumPy. NumPy by itself is a fairly low-level tool, similar to MATLAB. pandas on the other hand provides rich time series functionality, data alignment, NA-friendly statistics, groupby, merge and join methods, and lots of other conveniences. It has become very popular in recent years in financial applications. I will have a chapter dedicated to financial data analysis using pandas in my upcoming book.


Question:
Update row values where certain condition is met in pandas Say I have the following dataframe: What is the most efficient way to update the values of the columns feat and another_feat where the stream is number 2? Is this it?
```py
for index, row in df.iterrows():
    if df1.loc[index,'stream'] == 2:
       # do something
```
How do I do it if there are more than 100 columns? I don't want to explicitly name the columns that I want to update. I want to divide the value of each column by 2 (except for the stream column). So to be clear, my goal is: Dividing all values by 2 of all rows that have stream 2, but not changing the stream column.

Response:
I think you can use loc if you need update two columns to same value:
```
df1.loc[df1['stream'] == 2, ['feat','another_feat']] = 'aaaa'
print df1
   stream        feat another_feat
a       1  some_value   some_value
b       2        aaaa         aaaa
c       2        aaaa         aaaa
d       3  some_value   some_value
```
If you need update separate, one option is use:
```
df1.loc[df1['stream'] == 2, 'feat'] = 10
print df1
   stream        feat another_feat
a       1  some_value   some_value
b       2          10   some_value
c       2          10   some_value
d       3  some_value   some_value
```
Another common option is use numpy.where:
```
df1['feat'] = np.where(df1['stream'] == 2, 10,20)
print df1
   stream  feat another_feat
a       1    20   some_value
b       2    10   some_value
c       2    10   some_value
d       3    20   some_value
```
EDIT: If you need divide all columns without stream where condition is True, use:
```
print df1
   stream  feat  another_feat
a       1     4             5
b       2     4             5
c       2     2             9
d       3     1             7

#filter columns all without stream
cols = [col for col in df1.columns if col != 'stream']
print cols
['feat', 'another_feat']

df1.loc[df1['stream'] == 2, cols ] = df1 / 2
print df1
   stream  feat  another_feat
a       1   4.0           5.0
b       2   2.0           2.5
c       2   1.0           4.5
d       3   1.0           7.0
```
If working with multiple conditions is possible use multiple numpy.where or numpy.select:
```
df0 = pd.DataFrame({'Col':[5,0,-6]})

df0['New Col1'] = np.where((df0['Col'] > 0), 'Increasing',
                          np.where((df0['Col'] < 0), 'Decreasing', 'No Change'))

df0['New Col2'] = np.select([df0['Col'] > 0, df0['Col'] < 0],
                            ['Increasing',  'Decreasing'],
                            default='No Change')

print (df0)
   Col    New Col1    New Col2
0    5  Increasing  Increasing
1    0   No Change   No Change
2   -6  Decreasing  Decreasing
```


Question:
GroupBy pandas DataFrame and select most common value I have a data frame with three string columns. I know that the only one value in the 3rd column is valid for every combination of the first two. To clean the data I have to group by data frame by first two columns and select most common value of the third column for each combination. My code:
```
import pandas as pd
from scipy import stats

source = pd.DataFrame({
    'Country': ['USA', 'USA', 'Russia', 'USA'],
    'City': ['New-York', 'New-York', 'Sankt-Petersburg', 'New-York'],
    'Short name': ['NY', 'New', 'Spb', 'NY']})

source.groupby(['Country','City']).agg(lambda x: stats.mode(x['Short name'])[0])
```
Last line of code doesn't work, it says KeyError: 'Short name' and if I try to group only by City, then I got an AssertionError. What can I do fix it?

Response:
Pandas >= 0.16 pd.Series.mode is available! Use groupby, GroupBy.agg, and apply the pd.Series.mode function to each group:
```
source.groupby(['Country','City'])['Short name'].agg(pd.Series.mode)

Country  City
Russia   Sankt-Petersburg    Spb
USA      New-York             NY
Name: Short name, dtype: object
```
If this is needed as a DataFrame, use
```
source.groupby(['Country','City'])['Short name'].agg(pd.Series.mode).to_frame()

                         Short name
Country City
Russia  Sankt-Petersburg        Spb
USA     New-York                 NY
```
The useful thing about Series.mode is that it always returns a Series, making it very compatible with agg and apply, especially when reconstructing the groupby output. It is also faster.
```
# Accepted answer.
%timeit source.groupby(['Country','City']).agg(lambda x:x.value_counts().index[0])
# Proposed in this post.
%timeit source.groupby(['Country','City'])['Short name'].agg(pd.Series.mode)

5.56 ms ± 343 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
2.76 ms ± 387 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
Dealing with Multiple Modes Series.mode also does a good job when there are multiple modes:
```
source2 = source.append(
    pd.Series({'Country': 'USA', 'City': 'New-York', 'Short name': 'New'}),
    ignore_index=True)

# Now `source2` has two modes for the
# ("USA", "New-York") group, they are "NY" and "New".
source2

  Country              City Short name
0     USA          New-York         NY
1     USA          New-York        New
2  Russia  Sankt-Petersburg        Spb
3     USA          New-York         NY
4     USA          New-York        New
```
```
source2.groupby(['Country','City'])['Short name'].agg(pd.Series.mode)

Country  City
Russia   Sankt-Petersburg          Spb
USA      New-York            [NY, New]
Name: Short name, dtype: object
```
Or, if you want a separate row for each mode, you can use GroupBy.apply:
```
source2.groupby(['Country','City'])['Short name'].apply(pd.Series.mode)

Country  City
Russia   Sankt-Petersburg  0    Spb
USA      New-York          0     NY
                           1    New
Name: Short name, dtype: object
```
If you don't care which mode is returned as long as it's either one of them, then you will need a lambda that calls mode and extracts the first result.
```
source2.groupby(['Country','City'])['Short name'].agg(
    lambda x: pd.Series.mode(x)[0])

Country  City
Russia   Sankt-Petersburg    Spb
USA      New-York             NY
Name: Short name, dtype: object
```
Alternatives to (not) consider You can also use statistics.mode from python, but...
```
source.groupby(['Country','City'])['Short name'].apply(statistics.mode)

Country  City
Russia   Sankt-Petersburg    Spb
USA      New-York             NY
Name: Short name, dtype: object
```
...it does not work well when having to deal with multiple modes; a StatisticsError is raised. This is mentioned in the docs: If data is empty, or if there is not exactly one most common value, StatisticsError is raised. But you can see for yourself...
```
statistics.mode([1, 2])
# ---------------------------------------------------------------------------
# StatisticsError                           Traceback (most recent call last)
# ...
# StatisticsError: no unique mode; found 2 equally common values
```


Question:
How to merge multiple dataframes I have different dataframes and need to merge them together based on the date column. If I only had two dataframes, I could use df1.merge(df2, on='date'), to do it with three dataframes, I use df1.merge(df2.merge(df3, on='date'), on='date'), however it becomes really complex and unreadable to do it with multiple dataframes. All dataframes have one column in common -date, but they don't have the same number of rows nor columns and I only need those rows in which each date is common to every dataframe. So, I'm trying to write a recursion function that returns a dataframe with all data but it didn't work. How should I merge multiple dataframes then? I tried different ways and got errors like out of range, keyerror 0/1/2/3 and can not merge DataFrame with instance of type . This is the script I wrote:
```
dfs = [df1, df2, df3] # list of dataframes

def mergefiles(dfs, countfiles, i=0):
    if i == (countfiles - 2): # it gets to the second to last and merges it with the last
        return

    dfm = dfs[i].merge(mergefiles(dfs[i+1], countfiles, i=i+1), on='date')
    return dfm

print(mergefiles(dfs, len(dfs)))
```
An example: df_1:
```
May 19, 2017;1,200.00;0.1%
May 18, 2017;1,100.00;0.1%
May 17, 2017;1,000.00;0.1%
May 15, 2017;1,901.00;0.1%
```
df_2:
```
May 20, 2017;2,200.00;1000000;0.2%
May 18, 2017;2,100.00;1590000;0.2%
May 16, 2017;2,000.00;1230000;0.2%
May 15, 2017;2,902.00;1000000;0.2%
```
df_3:
```
May 21, 2017;3,200.00;2000000;0.3%
May 17, 2017;3,100.00;2590000;0.3%
May 16, 2017;3,000.00;2230000;0.3%
May 15, 2017;3,903.00;2000000;0.3%
```
Expected merge result:
```
May 15, 2017;  1,901.00;0.1%;  2,902.00;1000000;0.2%;   3,903.00;2000000;0.3%
```

Response:
Short answer
```py
df_merged = reduce(lambda  left,right: pd.merge(left,right,on=['DATE'],
                                            how='outer'), data_frames)
```
Long answer Below, is the most clean, comprehensible way of merging multiple dataframe if complex queries aren't involved. Just simply merge with DATE as the index and merge using OUTER method (to get all the data).
```
import pandas as pd
from functools import reduce

df1 = pd.read_table('file1.csv', sep=',')
df2 = pd.read_table('file2.csv', sep=',')
df3 = pd.read_table('file3.csv', sep=',')
```
Now, basically load all the files you have as data frame into a list. And, then merge the files using merge or reduce function.
```
# compile the list of dataframes you want to merge
data_frames = [df1, df2, df3]
```
Note: you can add as many data-frames inside the above list. This is the good part about this method. No complex queries involved. To keep the values that belong to the same date you need to merge it on the DATE
```
df_merged = reduce(lambda  left,right: pd.merge(left,right,on=['DATE'],
                                            how='outer'), data_frames)

# if you want to fill the values that don't exist in the lines of merged dataframe simply fill with required strings as

df_merged = reduce(lambda  left,right: pd.merge(left,right,on=['DATE'],
                                            how='outer'), data_frames).fillna('void')
```
Now, the output will the values from the same date on the same lines. You can fill the non existing data from different frames for different columns using fillna(). Then write the merged data to the csv file if desired.
```
pd.DataFrame.to_csv(df_merged, 'merged.txt', sep=',', na_rep='.', index=False)
```
This should give you DATE VALUE1 VALUE2 VALUE3 ....


Question:
Add column with constant value to pandas dataframe [duplicate] This question already has answers here: Add column to dataframe with constant value (10 answers) Closed 5 years ago. Given a DataFrame:
```
np.random.seed(0)
df = pd.DataFrame(np.random.randn(3, 3), columns=list('ABC'), index=[1, 2, 3])
df

          A         B         C
1  1.764052  0.400157  0.978738
2  2.240893  1.867558 -0.977278
3  0.950088 -0.151357 -0.103219
```
What is the simplest way to add a new column containing a constant value eg 0?
```
A         B         C  new
1  1.764052  0.400157  0.978738    0
2  2.240893  1.867558 -0.977278    0
3  0.950088 -0.151357 -0.103219    0
```
This is my solution, but I don't know why this puts NaN into 'new' column?
```
df['new'] = pd.Series([0 for x in range(len(df.index))])

          A         B         C  new
1  1.764052  0.400157  0.978738  0.0
2  2.240893  1.867558 -0.977278  0.0
3  0.950088 -0.151357 -0.103219  NaN
```

Response:
Super simple in-place assignment: df['new'] = 0 For in-place modification, perform direct assignment. This assignment is broadcasted by pandas for each row.
```
df = pd.DataFrame('x', index=range(4), columns=list('ABC'))
df

   A  B  C
0  x  x  x
1  x  x  x
2  x  x  x
3  x  x  x
```
```
df['new'] = 'y'
# Same as,
# df.loc[:, 'new'] = 'y'
df

   A  B  C new
0  x  x  x   y
1  x  x  x   y
2  x  x  x   y
3  x  x  x   y
```
Note for object columns If you want to add an column of empty lists, here is my advice: Consider not doing this. object columns are bad news in terms of performance. Rethink how your data is structured. Consider storing your data in a sparse data structure. More information: sparse data structures If you must store a column of lists, ensure not to copy the same reference multiple times.
```
# Wrong
df['new'] = [[]] * len(df)
# Right
df['new'] = [[] for _ in range(len(df))]
```
Generating a copy: df.assign(new=0) If you need a copy instead, use DataFrame.assign:
```
df.assign(new='y')

   A  B  C new
0  x  x  x   y
1  x  x  x   y
2  x  x  x   y
3  x  x  x   y
```
And, if you need to assign multiple such columns with the same value, this is as simple as,
```
c = ['new1', 'new2', ...]
df.assign(**dict.fromkeys(c, 'y'))

   A  B  C new1 new2
0  x  x  x    y    y
1  x  x  x    y    y
2  x  x  x    y    y
3  x  x  x    y    y
```
Multiple column assignment Finally, if you need to assign multiple columns with different values, you can use assign with a dictionary.
```
c = {'new1': 'w', 'new2': 'y', 'new3': 'z'}
df.assign(**c)

   A  B  C new1 new2 new3
0  x  x  x    w    y    z
1  x  x  x    w    y    z
2  x  x  x    w    y    z
3  x  x  x    w    y    z
```


Question:
Split a large pandas dataframe I have a large dataframe with 423244 lines. I want to split this in to 4. I tried the following code which gave an error? ValueError: array split does not result in an equal division
```
for item in np.split(df, 4):
    print item
```
How to split this dataframe in to 4 groups?

Response:
Use np.array_split:
```
Docstring:
Split an array into multiple sub-arrays.

Please refer to the ``split`` documentation.  The only difference
between these functions is that ``array_split`` allows
`indices_or_sections` to be an integer that does *not* equally
divide the axis.
```
```

```python
import pandas as pd
```

```python
df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
   ...:                           'foo', 'bar', 'foo', 'foo'],
   ...:                    'B' : ['one', 'one', 'two', 'three',
   ...:                           'two', 'two', 'one', 'three'],
   ...:                    'C' : randn(8), 'D' : randn(8)})
```

```python
print df
     A      B         C         D
0  foo    one -0.174067 -0.608579
1  bar    one -0.860386 -1.210518
2  foo    two  0.614102  1.689837
3  bar  three -0.284792 -1.071160
4  foo    two  0.843610  0.803712
5  bar    two -1.514722  0.870861
6  foo    one  0.131529 -0.968151
7  foo  three -1.002946 -0.257468
```

```python
import numpy as np
```

```python
np.array_split(df, 3)
#Output
#[     A    B         C         D
#0  foo  one -0.174067 -0.608579
#1  bar  one -0.860386 -1.210518
#2  foo  two  0.614102  1.689837,
#      A      B         C         D
#3  bar  three -0.284792 -1.071160
#4  foo    two  0.843610  0.803712
#5  bar    two -1.514722  0.870861,
#      A      B         C         D
#6  foo    one  0.131529 -0.968151
#7  foo  three -1.002946 -0.257468]
#```
```


Question:
Replace NaN in one column with value from corresponding row of second column I am working with this Pandas DataFrame in Python.
```
File    heat    Farheit Temp_Rating
   1    YesQ         75         N/A
   1    NoR         115         N/A
   1    YesA         63         N/A
   1    NoT          83          41
   1    NoY         100          80
   1    YesZ         56          12
   2    YesQ        111         N/A
   2    NoR          60         N/A
   2    YesA         19         N/A
   2    NoT         106          77
   2    NoY          45          21
   2    YesZ         40          54
   3    YesQ         84         N/A
   3    NoR          67         N/A
   3    YesA         94         N/A
   3    NoT          68          39
   3    NoY          63          46
   3    YesZ         34          81
```
I need to replace all NaNs in the Temp_Rating column with the value from the Farheit column. This is what I need:
```
File        heat    Temp_Rating
   1        YesQ             75
   1         NoR            115
   1        YesA             63
   1        YesQ             41
   1         NoR             80
   1        YesA             12
   2        YesQ            111
   2         NoR             60
   2        YesA             19
   2         NoT             77
   2         NoY             21
   2        YesZ             54
   3        YesQ             84
   3         NoR             67
   3        YesA             94
   3         NoT             39
   3         NoY             46
   3        YesZ             81
```
If I do a Boolean selection, I can pick out only one of these columns at a time. The problem is if I then try to join them, I am not able to do this while preserving the correct order. How can I only find Temp_Rating rows with the NaNs and replace them with the value in the same row of the Farheit column?

Response:
Assuming your DataFrame is in df:
```
df.Temp_Rating.fillna(df.Farheit, inplace=True)
del df['Farheit']
df.columns = 'File heat Observations'.split()
```
First replace any NaN values with the corresponding value of df.Farheit. Delete the 'Farheit' column. Then rename the columns. Here's the resulting DataFrame:
```
File  heat  Observations
0      1  YesQ            75
1      1   NoR           115
2      1  YesA            63
3      1   NoT            41
4      1   NoY            80
5      1  YesZ            12
6      2  YesQ           111
7      2   NoR            60
8      2  YesA            19
9      2   NoT            77
10     2   NoY            21
11     2  YesZ            54
12     3  YesQ            84
13     3   NoR            67
14     3  YesA            94
15     3   NoT            39
16     3   NoY            46
17     3  YesZ            81
```


Question:
How do you filter pandas dataframes by multiple columns? To filter a DataFrame (df) by a single column, if we consider data with male and females we might:
```
males = df[df[Gender]=='Male']
```
Question 1: But what if the data spanned multiple years and I wanted to only see males for 2014? In other languages I might do something like:
```
if A = "Male" and if B = "2014" then
```
(except I want to do this and get a subset of the original DataFrame in a new dataframe object) Question 2: How do I do this in a loop, and create a dataframe object for each unique sets of year and gender (i.e. a df for: 2013-Male, 2013-Female, 2014-Male, and 2014-Female?
```
for y in year:

for g in gender:

df = .....
```

Response:
Using & operator, don't forget to wrap the sub-statements with ():
```
males = df[(df[Gender]=='Male') & (df[Year]==2014)]
```
To store your DataFrames in a dict using a for loop:
```
from collections import defaultdict
dic={}
for g in ['male', 'female']:
    dic[g]=defaultdict(dict)
    for y in [2013, 2014]:
        dic[g][y]=df[(df[Gender]==g) & (df[Year]==y)] #store the DataFrames to a dict of dict
```
A demo for your getDF:
```
def getDF(dic, gender, year):
    return dic[gender][year]

print genDF(dic, 'male', 2014)
```


Question:
dropping rows from dataframe based on a "not in" condition [duplicate] This question already has answers here: How to filter Pandas dataframe using 'in' and 'not in' like in SQL (12 answers) Closed 5 years ago. I want to drop rows from a pandas dataframe when the value of the date column is in a list of dates. The following code doesn't work:
```
a=['2015-01-01' , '2015-02-01']

df=df[df.datecolumn not in a]
```
I get the following error: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Response:
You can use pandas.Dataframe.isin. pandas.Dateframe.isin will return boolean values depending on whether each element is inside the list a or not. You then invert this with the ~ to convert True to False and vice versa.
```
import pandas as pd

a = ['2015-01-01' , '2015-02-01']

df = pd.DataFrame(data={'date':['2015-01-01' , '2015-02-01', '2015-03-01' , '2015-04-01', '2015-05-01' , '2015-06-01']})

print(df)
#         date
#0  2015-01-01
#1  2015-02-01
#2  2015-03-01
#3  2015-04-01
#4  2015-05-01
#5  2015-06-01

df = df[~df['date'].isin(a)]

print(df)
#         date
#2  2015-03-01
#3  2015-04-01
#4  2015-05-01
#5  2015-06-01
```


Question:
Pandas Merge - How to avoid duplicating columns I am attempting a merge between two data frames. Each data frame has two index levels (date, cusip). In the columns, some columns match between the two (currency, adj date) for example. What is the best way to merge these by index, but to not take two copies of currency and adj date. Each data frame is 90 columns, so I am trying to avoid writing everything out by hand.
```
df:                 currency  adj_date   data_col1 ...
date        cusip
2012-01-01  XSDP      USD      2012-01-03   0.45
...

df2:                currency  adj_date   data_col2 ...
date        cusip
2012-01-01  XSDP      USD      2012-01-03   0.45
...
```
If I do:
```
dfNew = merge(df, df2, left_index=True, right_index=True, how='outer')
```
I get
```
dfNew:              currency_x  adj_date_x   data_col2 ... currency_y adj_date_y
date        cusip
2012-01-01  XSDP      USD      2012-01-03   0.45             USD         2012-01-03
```
Thank you! ...

Response:
You can work out the columns that are only in one DataFrame and use this to select a subset of columns in the merge.
```
cols_to_use = df2.columns.difference(df.columns)
```
Then perform the merge (note this is an index object but it has a handy tolist() method).
```
dfNew = merge(df, df2[cols_to_use], left_index=True, right_index=True, how='outer')
```
This will avoid any columns clashing in the merge.


Question:
Replace None with NaN in pandas dataframe I have table x:
```
website
0   http://www.google.com/
1   http://www.yahoo.com
2   None
```
I want to replace python None with pandas NaN. I tried:
```
x.replace(to_replace=None, value=np.nan)
```
But I got:
```
TypeError: 'regex' must be a string or a compiled regular expression or a list or dict of strings or regular expressions, you passed a 'bool'
```
How should I go about it?

Response:
You can use DataFrame.fillna or Series.fillna which will replace the Python object None, not the string 'None'.
```
import pandas as pd
import numpy as np
```
For dataframe:
```
df = df.fillna(value=np.nan)
```
For column or series:
```
df.mycol.fillna(value=np.nan, inplace=True)
```


Question:
Python pandas: fill a dataframe row by row The simple task of adding a row to a pandas.DataFrame object seems to be hard to accomplish. There are 3 stackoverflow questions relating to this, none of which give a working answer. Here is what I'm trying to do. I have a DataFrame of which I already know the shape as well as the names of the rows and columns.
```

```python
df = pandas.DataFrame(columns=['a','b','c','d'], index=['x','y','z'])
```

```python
df
#Output
#     a    b    c    d
#x  NaN  NaN  NaN  NaN
#y  NaN  NaN  NaN  NaN
#z  NaN  NaN  NaN  NaN
#```
#Now, I have a function to compute the values of the rows iteratively. How can I fill in one of the rows with either a dictionary or a pandas.Series ? Here are various attempts that have failed:
#```
```

```python
y = {'a':1, 'b':5, 'c':2, 'd':3}
```

```python
df['y'] = y
#Output
#AssertionError: Length of values does not match length of index
#```
#Apparently it tried to add a column instead of a row.
#```
```

```python
y = {'a':1, 'b':5, 'c':2, 'd':3}
```

```python
df.join(y)
#Output
#AttributeError: 'builtin_function_or_method' object has no attribute 'is_unique'
#```
#Very uninformative error message.
#```
```

```python
y = {'a':1, 'b':5, 'c':2, 'd':3}
```

```python
df.set_value(index='y', value=y)
#Output
#TypeError: set_value() takes exactly 4 arguments (3 given)
#```
#Apparently that is only for setting individual values in the dataframe.
#```
```

```python
y = {'a':1, 'b':5, 'c':2, 'd':3}
```

```python
df.append(y)
#Output
#Exception: Can only append a Series if ignore_index=True
#```
#Well, I don't want to ignore the index, otherwise here is the result:
#```
```

```python
df.append(y, ignore_index=True)
#Output
#     a    b    c    d
#0  NaN  NaN  NaN  NaN
#1  NaN  NaN  NaN  NaN
#2  NaN  NaN  NaN  NaN
#3    1    5    2    3
#```
#It did align the column names with the values, but lost the row labels.
#```
```

```python
y = {'a':1, 'b':5, 'c':2, 'd':3}
```

```python
df.ix['y'] = y
```

```python
df
#Output
#                                  a                                 b  \
#x                               NaN                               NaN
#y  {'a': 1, 'c': 2, 'b': 5, 'd': 3}  {'a': 1, 'c': 2, 'b': 5, 'd': 3}
#z                               NaN                               NaN
```

c                                 d
x                               NaN                               NaN
y  {'a': 1, 'c': 2, 'b': 5, 'd': 3}  {'a': 1, 'c': 2, 'b': 5, 'd': 3}
z                               NaN                               NaN
```
That also failed miserably. So how do you do it ?

Response:
df['y'] will set a column since you want to set a row, use .loc Note that .ix is equivalent here, yours failed because you tried to assign a dictionary to each element of the row y probably not what you want; converting to a Series tells pandas that you want to align the input (for example you then don't have to to specify all of the elements)
```

```python
import pandas as pd
```

```python
df = pd.DataFrame(columns=['a','b','c','d'], index=['x','y','z'])
```

```python
df.loc['y'] = pd.Series({'a':1, 'b':5, 'c':2, 'd':3})
```

```python
df
#Output
#a    b    c    d
#x  NaN  NaN  NaN  NaN
#y    1    5    2    3
#z  NaN  NaN  NaN  NaN
#```
```


Question:
how to check the dtype of a column in python pandas I need to use different functions to treat numeric columns and string columns. What I am doing now is really dumb:
```
allc = list((agg.loc[:, (agg.dtypes==np.float64)|(agg.dtypes==np.int)]).columns)
for y in allc:
    treat_numeric(agg[y])

allc = list((agg.loc[:, (agg.dtypes!=np.float64)&(agg.dtypes!=np.int)]).columns)
for y in allc:
    treat_str(agg[y])
```
Is there a more elegant way to do this? E.g.
```
for y in agg.columns:
    if(dtype(agg[y]) == 'string'):
          treat_str(agg[y])
    elif(dtype(agg[y]) != 'string'):
          treat_numeric(agg[y])
```

Response:
In pandas 0.20.2 you can do:
```
from pandas.api.types import is_string_dtype
from pandas.api.types import is_numeric_dtype

is_string_dtype(df['A'])
>>>> True

is_numeric_dtype(df['B'])
>>>> True
```
So your code becomes:
```
for y in agg.columns:
    if (is_string_dtype(agg[y])):
        treat_str(agg[y])
    elif (is_numeric_dtype(agg[y])):
        treat_numeric(agg[y])
```


Question:
Move column by name to front of table in pandas Here is my df:
```
Net   Upper   Lower  Mid  Zsore
Answer option
More than once a day          0%   0.22%  -0.12%   2    65
Once a day                    0%   0.32%  -0.19%   3    45
Several times a week          2%   2.45%   1.10%   4    78
Once a week                   1%   1.63%  -0.40%   6    65
```
How can I move a column by name ("Mid") to the front of the table, index 0. This is what the result should look like:
```
Mid   Upper   Lower  Net  Zsore
Answer option
More than once a day          2   0.22%  -0.12%   0%    65
Once a day                    3   0.32%  -0.19%   0%    45
Several times a week          4   2.45%   1.10%   2%    78
Once a week                   6   1.63%  -0.40%   1%    65
```
My current code moves the column by index using df.columns.tolist() but I'd like to shift it by name.

Response:
We can use loc to reorder by passing a list:
```

```python
# get a list of columns
cols = list(df)
# move the column to head of list using index, pop and insert
cols.insert(0, cols.pop(cols.index('Mid')))
cols
#Output
#['Mid', 'Net', 'Upper', 'Lower', 'Zsore']
```

```python
# use ix to reorder
df = df.loc[:, cols]
df
#Output
#Mid Net  Upper   Lower  Zsore
#Answer_option
#More_than_once_a_day    2  0%  0.22%  -0.12%     65
#Once_a_day              3  0%  0.32%  -0.19%     45
#Several_times_a_week    4  2%  2.45%   1.10%     78
#Once_a_week             6  1%  1.63%  -0.40%     65
#```
#Another method is to take a reference to the column and reinsert it at the front:
#```
```

```python
mid = df['Mid']
df.drop(labels=['Mid'], axis=1,inplace = True)
df.insert(0, 'Mid', mid)
df
#Output
#Mid Net  Upper   Lower  Zsore
#Answer_option
#More_than_once_a_day    2  0%  0.22%  -0.12%     65
#Once_a_day              3  0%  0.32%  -0.19%     45
#Several_times_a_week    4  2%  2.45%   1.10%     78
#Once_a_week             6  1%  1.63%  -0.40%     65
#```
#You can, with very early versions of Pandas, also use ix to achieve the same results:
#```
#df = df.ix[:, cols]
#```
#But ix was deprecated from pandas 0.20.0 onwards and was discontinued as of Pandas 1.0.
```


Question:
Get pandas.read_csv to read empty values as empty string instead of nan I'm using the pandas library to read in some CSV data. In my data, certain columns contain strings. The string "nan" is a possible value, as is an empty string. I managed to get pandas to read "nan" as a string, but I can't figure out how to get it not to read an empty value as NaN. Here's sample data and output
```
One,Two,Three
a,1,one
b,2,two
,3,three
d,4,nan
e,5,five
nan,6,
g,7,seven

```python
pandas.read_csv('test.csv', na_values={'One': [], "Three": []})
#Output
#    One  Two  Three
#0    a    1    one
#1    b    2    two
#2  NaN    3  three
#3    d    4    nan
#4    e    5   five
#5  nan    6    NaN
#6    g    7  seven
#```
#It correctly reads "nan" as the string "nan', but still reads the empty cells as NaN. I tried passing in str in the converters argument to read_csv (with converters={'One': str})), but it still reads the empty cells as NaN. I realize I can fill the values after reading, with fillna, but is there really no way to tell pandas that an empty cell in a particular CSV column should be read as an empty string instead of NaN?
```

Response:
I was still confused after reading the other answers and comments. But the answer now seems simpler, so here you go. Since Pandas version 0.9 (from 2012), you can read your csv with empty cells interpreted as empty strings by simply setting keep_default_na=False:
```
pd.read_csv('test.csv', keep_default_na=False)
```
This issue is more clearly explained in More consistent na_values handling in read_csv · Issue #1657 · pandas-dev/pandas That was fixed on on Aug 19, 2012 for Pandas version 0.9 in BUG: more consistent na_values #1657 · pandas-dev/pandas@d9abf68


Question:
How to create a DataFrame of random integers with Pandas? I know that if I use randn, the following code gives me what I am looking for, but with elements from a normal distribution. But what if I just wanted random integers?
```py
import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randn(100, 4), columns=list('ABCD'))
```
randint works by providing a range, but not an array like randn. So how do I do this with random integers between some range?

Response:
numpy.random.randint accepts a third argument (size) , in which you can specify the size of the output array. You can use this to create your DataFrame -
```
df = pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))
```
Here - np.random.randint(0,100,size=(100, 4)) - creates an output array of size (100,4) with random integer elements between [0,100) . Demo -
```python
import numpy as np
import pandas as pd
df = pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))
```
which produces:
```none
A   B   C   D
0   45  88  44  92
1   62  34   2  86
2   85  65  11  31
3   74  43  42  56
4   90  38  34  93
5    0  94  45  10
6   58  23  23  60
..  ..  ..  ..  ..
```


Question:
Use .corr to get the correlation between two columns I have the following pandas dataframe Top15: I create a column that estimates the number of citable documents per person:
```
Top15['PopEst'] = Top15['Energy Supply'] / Top15['Energy Supply per Capita']
Top15['Citable docs per Capita'] = Top15['Citable documents'] / Top15['PopEst']
```
I want to know the correlation between the number of citable documents per capita and the energy supply per capita. So I use the .corr() method (Pearson's correlation):
```
data = Top15[['Citable docs per Capita','Energy Supply per Capita']]
correlation = data.corr(method='pearson')
```
I want to return a single number, but the result is:

Response:
Without actual data it is hard to answer the question but I guess you are looking for something like this:
```
Top15['Citable docs per Capita'].corr(Top15['Energy Supply per Capita'])
```
That calculates the correlation between your two columns 'Citable docs per Capita' and 'Energy Supply per Capita'. To give an example:
```
import pandas as pd

df = pd.DataFrame({'A': range(4), 'B': [2*i for i in range(4)]})

   A  B
0  0  0
1  1  2
2  2  4
3  3  6
```
Then
```
df['A'].corr(df['B'])
```
gives 1 as expected. Now, if you change a value, e.g.
```
df.loc[2, 'B'] = 4.5

   A    B
0  0  0.0
1  1  2.0
2  2  4.5
3  3  6.0
```
the command
```
df['A'].corr(df['B'])
```
returns
```
0.99586
```
which is still close to 1, as expected. If you apply .corr() directly to your dataframe, it will return all pairwise correlations between your columns; that's why you then observe 1s at the diagonal of your matrix (each column is perfectly correlated with itself).
```
df.corr()
```
will therefore return
```
A         B
A  1.000000  0.995862
B  0.995862  1.000000
```
In the graphic you show, only the upper left corner of the correlation matrix is represented (I assume). There can be cases, where you get NaNs in your solution - check this post for an example. If you want to filter entries above/below a certain threshold, you can check this question. If you want to plot a heatmap of the correlation coefficients, you can check this answer and if you then run into the issue with overlapping axis-labels check the following post.


Question:
How to select all columns whose names start with X in a pandas DataFrame I have a DataFrame:
```
import pandas as pd
import numpy as np

df = pd.DataFrame({'foo.aa': [1, 2.1, np.nan, 4.7, 5.6, 6.8],
                   'foo.fighters': [0, 1, np.nan, 0, 0, 0],
                   'foo.bars': [0, 0, 0, 0, 0, 1],
                   'bar.baz': [5, 5, 6, 5, 5.6, 6.8],
                   'foo.fox': [2, 4, 1, 0, 0, 5],
                   'nas.foo': ['NA', 0, 1, 0, 0, 0],
                   'foo.manchu': ['NA', 0, 0, 0, 0, 0],})
```
I want to select values of 1 in columns starting with foo.. Is there a better way to do it other than:
```
df2 = df[(df['foo.aa'] == 1)|
(df['foo.fighters'] == 1)|
(df['foo.bars'] == 1)|
(df['foo.fox'] == 1)|
(df['foo.manchu'] == 1)
]
```
Something similar to writing something like:
```
df2= df[df.STARTS_WITH_FOO == 1]
```
The answer should print out a DataFrame like this:
```
bar.baz  foo.aa  foo.bars  foo.fighters  foo.fox foo.manchu nas.foo
0      5.0     1.0         0             0        2         NA      NA
1      5.0     2.1         0             1        4          0       0
2      6.0     NaN         0           NaN        1          0       1
5      6.8     6.8         1             0        5          0       0

[4 rows x 7 columns]
```

Response:
Just perform a list comprehension to create your columns:
```

```python
filter_col = [col for col in df if col.startswith('foo')]
filter_col
#Output
#['foo.aa', 'foo.bars', 'foo.fighters', 'foo.fox', 'foo.manchu']
```

```python
df[filter_col]
#Output
#foo.aa  foo.bars  foo.fighters  foo.fox foo.manchu
#0     1.0         0             0        2         NA
#1     2.1         0             1        4          0
#2     NaN         0           NaN        1          0
#3     4.7         0             0        0          0
#4     5.6         0             0        0          0
#5     6.8         1             0        5          0
#```
#Another method is to create a series from the columns and use the vectorised str method startswith:
#```
```

```python
df[df.columns[pd.Series(df.columns).str.startswith('foo')]]
#Output
#foo.aa  foo.bars  foo.fighters  foo.fox foo.manchu
#0     1.0         0             0        2         NA
#1     2.1         0             1        4          0
#2     NaN         0           NaN        1          0
#3     4.7         0             0        0          0
#4     5.6         0             0        0          0
#5     6.8         1             0        5          0
#```
#In order to achieve what you want you need to add the following to filter the values that don't meet your ==1 criteria:
#```
```

```python
df[df[df.columns[pd.Series(df.columns).str.startswith('foo')]]==1]
#Output
#bar.baz  foo.aa  foo.bars  foo.fighters  foo.fox foo.manchu nas.foo
#0      NaN       1       NaN           NaN      NaN        NaN     NaN
#1      NaN     NaN       NaN             1      NaN        NaN     NaN
#2      NaN     NaN       NaN           NaN        1        NaN     NaN
#3      NaN     NaN       NaN           NaN      NaN        NaN     NaN
#4      NaN     NaN       NaN           NaN      NaN        NaN     NaN
#5      NaN     NaN         1           NaN      NaN        NaN     NaN
#```
#EDIT OK after seeing what you want the convoluted answer is this:
#```
```

```python
df.loc[df[df[df.columns[pd.Series(df.columns).str.startswith('foo')]] == 1].dropna(how='all', axis=0).index]
#Output
#bar.baz  foo.aa  foo.bars  foo.fighters  foo.fox foo.manchu nas.foo
#0      5.0     1.0         0             0        2         NA      NA
#1      5.0     2.1         0             1        4          0       0
#2      6.0     NaN         0           NaN        1          0       1
#5      6.8     6.8         1             0        5          0       0
#```
```


Question:
What is the difference between using loc and using just square brackets to filter for columns in Pandas/Python? I've noticed three methods of selecting a column in a Pandas DataFrame: First method of selecting a column using loc:
```
df_new = df.loc[:, 'col1']
```
Second method - seems simpler and faster:
```
df_new = df['col1']
```
Third method - most convenient:
```
df_new = df.col1
```
Is there a difference between these three methods? I don't think so, in which case I'd rather use the third method. I'm mostly curious as to why there appear to be three methods for doing the same thing.

Response:
In the following situations, they behave the same: Selecting a single column (df['A'] is the same as df.loc[:, 'A'] -> selects column A) Selecting a list of columns (df[['A', 'B', 'C']] is the same as df.loc[:, ['A', 'B', 'C']] -> selects columns A, B and C) Slicing by rows (df[1:3] is the same as df.iloc[1:3] -> selects rows 1 and 2. Note, however, if you slice rows with loc, instead of iloc, you'll get rows 1, 2 and 3 assuming you have a RangeIndex. See details here.) However, [] does not work in the following situations: You can select a single row with df.loc[row_label] You can select a list of rows with df.loc[[row_label1, row_label2]] You can slice columns with df.loc[:, 'A':'C'] These three cannot be done with []. More importantly, if your selection involves both rows and columns, then assignment becomes problematic.
```
df[1:3]['A'] = 5
```
This selects rows 1 and 2 then selects column 'A' of the returning object and assigns value 5 to it. The problem is, the returning object might be a copy so this may not change the actual DataFrame. This raises SettingWithCopyWarning. The correct way of making this assignment is:
```
df.loc[1:3, 'A'] = 5
```
With .loc, you are guaranteed to modify the original DataFrame. It also allows you to slice columns (df.loc[:, 'C':'F']), select a single row (df.loc[5]), and select a list of rows (df.loc[[1, 2, 5]]). Also note that these two were not included in the API at the same time. .loc was added much later as a more powerful and explicit indexer. See unutbu's answer for more detail. Note: Getting columns with [] vs . is a completely different topic. . is only there for convenience. It only allows accessing columns whose names are valid Python identifiers (i.e. they cannot contain spaces, they cannot be composed of numbers...). It cannot be used when the names conflict with Series/DataFrame methods. It also cannot be used for non-existing columns (i.e. the assignment df.a = 1 won't work if there is no column a). Other than that, . and [] are the same.


Question: How to drop rows from pandas data frame that contains a particular string in a particular column? [duplicate] This question already has answers here: Search for "does-not-contain" on a DataFrame in pandas (11 answers) Closed 6 years ago. I have a very large data frame in python and I want to drop all rows that have a particular string inside a particular column. For example, I want to drop all rows which have the string "XYZ" as a substring in the column C of the data frame. Can this be implemented in an efficient way using .drop() method?

Response:
pandas has vectorized string operations, so you can just filter out the rows that contain the string you don't want:
```

```python
df = pd.DataFrame(dict(A=[5,3,5,6], C=["foo","bar","fooXYZbar", "bat"]))
```

```python
df
#Output
#A          C
#0  5        foo
#1  3        bar
#2  5  fooXYZbar
#3  6        bat
```

```python
df[~df.C.str.contains("XYZ")]
#Output
#A    C
#0  5  foo
#1  3  bar
#3  6  bat
#```
```


Question:
cartesian product in pandas I have two pandas dataframes:
```
from pandas import DataFrame
df1 = DataFrame({'col1':[1,2],'col2':[3,4]})
df2 = DataFrame({'col3':[5,6]})
```
What is the best practice to get their cartesian product (of course without writing it explicitly like me)?
```
#df1, df2 cartesian product
df_cartesian = DataFrame({'col1':[1,2,1,2],'col2':[3,4,3,4],'col3':[5,5,6,6]})
```

Response:
In recent versions of Pandas (>= 1.2) this is built into merge so you can do:
```
from pandas import DataFrame
df1 = DataFrame({'col1':[1,2],'col2':[3,4]})
df2 = DataFrame({'col3':[5,6]})

df1.merge(df2, how='cross')
```
This is equivalent to the previous pandas < 1.2 answer but is easier to read. For pandas < 1.2: If you have a key that is repeated for each row, then you can produce a cartesian product using merge (like you would in SQL).
```
from pandas import DataFrame, merge
df1 = DataFrame({'key':[1,1], 'col1':[1,2],'col2':[3,4]})
df2 = DataFrame({'key':[1,1], 'col3':[5,6]})

merge(df1, df2,on='key')[['col1', 'col2', 'col3']]
```
Output:
```
col1  col2  col3
0     1     3     5
1     1     3     6
2     2     4     5
3     2     4     6
```
See here for the documentation: http://pandas.pydata.org/pandas-docs/stable/merging.html


Question:
How to rotate x-axis tick labels in a pandas plot With the following code:
```
import matplotlib
matplotlib.style.use('ggplot')
import matplotlib.pyplot as plt
import pandas as pd

df = pd.DataFrame({ 'celltype':["foo","bar","qux","woz"], 's1':[5,9,1,7], 's2':[12,90,13,87]})
df = df[["celltype","s1","s2"]]
df.set_index(["celltype"],inplace=True)
df.plot(kind='bar',alpha=0.75)
plt.xlabel("")
```
I made this plot: How can I rotate the x-axis tick labels to 0 degrees? I tried adding this but did not work:
```
plt.set_xticklabels(df.index,rotation=90)
```

Response:
Pass param rot=0 to rotate the xticklabels:
```
import matplotlib
matplotlib.style.use('ggplot')
import matplotlib.pyplot as plt
import pandas as pd

df = pd.DataFrame({ 'celltype':["foo","bar","qux","woz"], 's1':[5,9,1,7], 's2':[12,90,13,87]})
df = df[["celltype","s1","s2"]]
df.set_index(["celltype"],inplace=True)
df.plot(kind='bar',alpha=0.75, rot=0)
plt.xlabel("")
plt.show()
```
yields plot:


Question:
Give column name when read csv file pandas This is the example of my dataset.
```

```python
user1 = pd.read_csv('dataset/1.csv')
```

```python
print(user1)
#Output
#          0  0.69464   3.1735   7.5048
#0  0.030639  0.14982  3.48680   9.2755
#1  0.069763 -0.29965  1.94770   9.1120
#2  0.099823 -1.68890  1.41650  10.1200
#3  0.129820 -2.17930  0.95342  10.9240
#4  0.159790 -2.30180  0.23155  10.6510
#5  0.189820 -1.41650  1.18500  11.0730
#```
#How to push down the first column and add the names column [TIME, X, Y, and Z] on the first column. The desired output is like this:
#```
#TIME        X        Y        Z
#0         0  0.69464   3.1735   7.5048
#1  0.030639  0.14982  3.48680   9.2755
#2  0.069763 -0.29965  1.94770   9.1120
#3  0.099823 -1.68890  1.41650  10.1200
#4  0.129820 -2.17930  0.95342  10.9240
#5  0.159790 -2.30180  0.23155  10.6510
#6  0.189820 -1.41650  1.18500  11.0730
#```
```

Response:
I'd do it like this:
```
colnames=['TIME', 'X', 'Y', 'Z']
user1 = pd.read_csv('dataset/1.csv', names=colnames, header=None)
```


Question:
Drop rows containing empty cells from a pandas DataFrame I have a pd.DataFrame that was created by parsing some excel spreadsheets. A column of which has empty cells. For example, below is the output for the frequency of that column, 32320 records have missing values for Tenant.
```

```python
value_counts(Tenant, normalize=False)
#Output
#                              32320
#    Thunderhead                8170
#    Big Data Others            5700
#    Cloud Cruiser              5700
#    Partnerpedia               5700
#    Comcast                    5700
#    SDP                        5700
#    Agora                      5700
#    dtype: int64
#```
#I am trying to drop rows where Tenant is missing, however .isnull() option does not recognize the missing values.
#```
```

```python
df['Tenant'].isnull().sum()
#Output
#    0
#```
#The column has data type "Object". What is happening in this case? How can I drop records where Tenant is missing?
```

Response:
Pandas will recognise a value as null if it is a np.nan object, which will print as NaN in the DataFrame. Your missing values are probably empty strings, which Pandas doesn't recognise as null. To fix this, you can convert the empty stings (or whatever is in your empty cells) to np.nan objects using replace(), and then call dropna()on your DataFrame to delete rows with null tenants. To demonstrate, we create a DataFrame with some random values and some empty strings in a Tenants column:
```

```python
import pandas as pd
```

```python
import numpy as np
```

```python

```

```python
df = pd.DataFrame(np.random.randn(10, 2), columns=list('AB'))
```

```python
df['Tenant'] = np.random.choice(['Babar', 'Rataxes', ''], 10)
```

```python
print df
```

A         B   Tenant
0 -0.588412 -1.179306    Babar
1 -0.008562  0.725239
2  0.282146  0.421721  Rataxes
3  0.627611 -0.661126    Babar
4  0.805304 -0.834214
5 -0.514568  1.890647    Babar
6 -1.188436  0.294792  Rataxes
7  1.471766 -0.267807    Babar
8 -1.730745  1.358165  Rataxes
9  0.066946  0.375640
```
Now we replace any empty strings in the Tenants column with np.nan objects, like so:
```

```python
df['Tenant'].replace('', np.nan, inplace=True)
```

```python
print df
```

A         B   Tenant
0 -0.588412 -1.179306    Babar
1 -0.008562  0.725239      NaN
2  0.282146  0.421721  Rataxes
3  0.627611 -0.661126    Babar
4  0.805304 -0.834214      NaN
5 -0.514568  1.890647    Babar
6 -1.188436  0.294792  Rataxes
7  1.471766 -0.267807    Babar
8 -1.730745  1.358165  Rataxes
9  0.066946  0.375640      NaN
```
Now we can drop the null values:
```

```python
df.dropna(subset=['Tenant'], inplace=True)
```

```python
print df
```

A         B   Tenant
0 -0.588412 -1.179306    Babar
2  0.282146  0.421721  Rataxes
3  0.627611 -0.661126    Babar
5 -0.514568  1.890647    Babar
6 -1.188436  0.294792  Rataxes
7  1.471766 -0.267807    Babar
8 -1.730745  1.358165  Rataxes
```


Question:
How to add a suffix (or prefix) to each column name? I want to add _x suffix to each column name like so:
```
featuresA = myPandasDataFrame.columns.values + '_x'
```
How do I do this? Additionally, if I wanted to add x_ as a suffix, how would the solution change?

Response:
The following is the nicest way to add suffix in my opinion.
```
df = df.add_suffix('_some_suffix')
```
As it is a function that is called on DataFrame and returns DataFrame - you can use it in chain of the calls.


Question:
Pandas dataframe groupby datetime month Consider a CSV file:
```
string,date,number
a string,2/5/11 9:16am,1.0
a string,3/5/11 10:44pm,2.0
a string,4/22/11 12:07pm,3.0
a string,4/22/11 12:10pm,4.0
a string,4/29/11 11:59am,1.0
a string,5/2/11 1:41pm,2.0
a string,5/2/11 2:02pm,3.0
a string,5/2/11 2:56pm,4.0
a string,5/2/11 3:00pm,5.0
a string,5/2/14 3:02pm,6.0
a string,5/2/14 3:18pm,7.0
```
I can read this in, and reformat the date column into datetime format:
```
b = pd.read_csv('b.dat')
b['date'] = pd.to_datetime(b['date'],format='%m/%d/%y %I:%M%p')
```
I have been trying to group the data by month. It seems like there should be an obvious way of accessing the month and grouping by that. But I can't seem to do it. Does anyone know how? What I am currently trying is re-indexing by the date:
```
b.index = b['date']
```
I can access the month like so:
```
b.index.month
```
However I can't seem to find a function to lump together by month.

Response:
Managed to do it:
```
b = pd.read_csv('b.dat')
b.index = pd.to_datetime(b['date'],format='%m/%d/%y %I:%M%p')
b.groupby(by=[b.index.month, b.index.year])
```
Or
```
b.groupby(pd.Grouper(freq='M'))  # update for v0.21+
```


Question:
Assign pandas dataframe column dtypes I want to set the dtypes of multiple columns in pd.Dataframe (I have a file that I've had to manually parse into a list of lists, as the file was not amenable for pd.read_csv)
```
import pandas as pd
print pd.DataFrame([['a','1'],['b','2']],
                   dtype={'x':'object','y':'int'},
                   columns=['x','y'])
```
I get
```
ValueError: entry not a 2- or 3- tuple
```
The only way I can set them is by looping through each column variable and recasting with astype.
```
dtypes = {'x':'object','y':'int'}
mydata = pd.DataFrame([['a','1'],['b','2']],
                      columns=['x','y'])
for c in mydata.columns:
    mydata[c] = mydata[c].astype(dtypes[c])
print mydata['y'].dtype   #=> int64
```
Is there a better way?

Response:
Since 0.17, you have to use the explicit conversions:
```
pd.to_datetime, pd.to_timedelta and pd.to_numeric
```
(As mentioned below, no more "magic", convert_objects has been deprecated in 0.17)
```
df = pd.DataFrame({'x': {0: 'a', 1: 'b'}, 'y': {0: '1', 1: '2'}, 'z': {0: '2018-05-01', 1: '2018-05-02'}})

df.dtypes

x    object
y    object
z    object
dtype: object

df

   x  y           z
0  a  1  2018-05-01
1  b  2  2018-05-02
```
You can apply these to each column you want to convert:
```
df["y"] = pd.to_numeric(df["y"])
df["z"] = pd.to_datetime(df["z"])
df

   x  y          z
0  a  1 2018-05-01
1  b  2 2018-05-02

df.dtypes

x            object
y             int64
z    datetime64[ns]
dtype: object
```
and confirm the dtype is updated. OLD/DEPRECATED ANSWER for pandas 0.12 - 0.16: You can use convert_objects to infer better dtypes:
```

```python
df
#Output
#x  y
#0  a  1
#1  b  2
```

```python
df.dtypes
#Output
#x    object
#y    object
#dtype: object
```

```python
df.convert_objects(convert_numeric=True)
#Output
#x  y
#0  a  1
#1  b  2
```

```python
df.convert_objects(convert_numeric=True).dtypes
#Output
#x    object
#y     int64
#dtype: object
#```
#Magic! (Sad to see it deprecated.)
```


Question:
Pandas: change data type of Series to String I use Pandas 'ver 0.12.0' with Python 2.7 and have a dataframe as below:
```
df = pd.DataFrame({'id' : [123,512,'zhub1', 12354.3, 129, 753, 295, 610],
                    'colour': ['black', 'white','white','white',
                            'black', 'black', 'white', 'white'],
                    'shape': ['round', 'triangular', 'triangular','triangular','square',
                                        'triangular','round','triangular']
                    },  columns= ['id','colour', 'shape'])
```
The id Series consists of some integers and strings. Its dtype by default is object. I want to convert all contents of id to strings. I tried astype(str), which produces the output below.
```
df['id'].astype(str)
0    1
1    5
2    z
3    1
4    1
5    7
6    2
7    6
```
1) How can I convert all elements of id to String? 2) I will eventually use id for indexing for dataframes. Would having String indices in a dataframe slow things down, compared to having an integer index?

Response:
A new answer to reflect the most current practices: as of now (v1.2.4), neither astype('str') nor astype(str) work. As per the documentation, a Series can be converted to the string datatype in the following ways:
```
df['id'] = df['id'].astype("string")

df['id'] = pandas.Series(df['id'], dtype="string")

df['id'] = pandas.Series(df['id'], dtype=pandas.StringDtype)
```
End to end example:
```py
import pandas as pd

# Create a sample DataFrame
data = {
    'Name': ['John', 'Alice', 'Bob', 'John', 'Alice'],
    'Age': [25, 30, 35, 25, 30],
    'City': ['New York', 'London', 'Paris', 'New York', 'London'],
    'Salary': [50000, 60000, 70000, 50000, 60000],
    'Category': ['A', 'B', 'C', 'A', 'B']
}

df = pd.DataFrame(data)

# Print the DataFrame
print("Original DataFrame:")
print(df)
print("\nData types:")
print(df.dtypes)
cat_cols_ = None
# Apply the code to change data types
if not cat_cols_:
    # Get the columns with object data type
    object_columns = df.select_dtypes(include=['object']).columns.tolist()

    if len(object_columns) > 0:
        print(f"\nObject columns found, converting to string: {object_columns}")

        # Convert object columns to string type
        df[object_columns] = df[object_columns].astype('string')

    # Get the categorical columns (including string and category data types)
    cat_cols_ = df.select_dtypes(include=['category', 'string']).columns.tolist()

# Print the updated DataFrame and data types
print("\nUpdated DataFrame:")
print(df)
print("\nUpdated data types:")
print(df.dtypes)
print(f"\nCategorical columns (cat_cols_): {cat_cols_}")
```
```bash
Original DataFrame:
    Name  Age      City  Salary Category
0   John   25  New York   50000        A
1  Alice   30    London   60000        B
2    Bob   35     Paris   70000        C
3   John   25  New York   50000        A
4  Alice   30    London   60000        B

Data types:
Name        object
Age          int64
City        object
Salary       int64
Category    object
dtype: object

Object columns found, converting to string: ['Name', 'City', 'Category']

Updated DataFrame:
    Name  Age      City  Salary Category
0   John   25  New York   50000        A
1  Alice   30    London   60000        B
2    Bob   35     Paris   70000        C
3   John   25  New York   50000        A
4  Alice   30    London   60000        B

Updated data types:
Name        string[python]
Age                  int64
City        string[python]
Salary               int64
Category    string[python]
dtype: object

Categorical columns (cat_cols_): ['Name', 'City', 'Category']
```


Question:
Creating a new column based on if-elif-else condition [duplicate] This question already has answers here: Create Column with ELIF in Pandas (5 answers) Closed 1 year ago. I have a DataFrame df:
```
A    B
a   2    2
b   3    1
c   1    3
```
I want to create a new column based on the following criteria: if row A == B: 0 if rowA > B: 1 if row A < B: -1 so given the above table, it should be:
```
A    B    C
a   2    2    0
b   3    1    1
c   1    3   -1
```
For typical if else cases I do np.where(df.A > df.B, 1, -1), does pandas provide a special syntax for solving my problem with one step (without the necessity of creating 3 new columns and then combining the result)?

Response:
To formalize some of the approaches laid out above: Create a function that operates on the rows of your dataframe like so:
```
def f(row):
    if row['A'] == row['B']:
        val = 0
    elif row['A'] > row['B']:
        val = 1
    else:
        val = -1
    return val
```
Then apply it to your dataframe passing in the axis=1 option:
```

```python
df['C'] = df.apply(f, axis=1)
```

```python
df
#Output
#A  B  C
#a  2  2  0
#b  3  1  1
#c  1  3 -1
#```
#Of course, this is not vectorized so performance may not be as good when scaled to a large number of records. Still, I think it is much more readable. Especially coming from a SAS background. Edit Here is the vectorized version
#```
#df['C'] = np.where(
#    df['A'] == df['B'], 0, np.where(
#    df['A'] >  df['B'], 1, -1))
#```
```


Question:
Update index after sorting data-frame Take the following data-frame:
```
x = np.tile(np.arange(3),3)
y = np.repeat(np.arange(3),3)
df = pd.DataFrame({"x": x, "y": y})
```
```
x  y
0  0  0
1  1  0
2  2  0
3  0  1
4  1  1
5  2  1
6  0  2
7  1  2
8  2  2
```
I need to sort it by x first, and only second by y:
```
df2 = df.sort(["x", "y"])
```
```
x  y
0  0  0
3  0  1
6  0  2
1  1  0
4  1  1
7  1  2
2  2  0
5  2  1
8  2  2
```
How can I change the index such that it is ascending again. I.e. how do I get this:
```
x  y
0  0  0
1  0  1
2  0  2
3  1  0
4  1  1
5  1  2
6  2  0
7  2  1
8  2  2
```
I have tried the following. Unfortunately, it doesn't change the index at all:
```
df2.reindex(np.arange(len(df2.index)))
```

Response:
You can reset the index using reset_index to get back a default index of 0, 1, 2, ..., n-1 (and use drop=True to indicate you want to drop the existing index instead of adding it as an additional column to your dataframe):
```

```python
df2 = df2.reset_index(drop=True)
```

```python
df2
#Output
#x  y
#0  0  0
#1  0  1
#2  0  2
#3  1  0
#4  1  1
#5  1  2
#6  2  0
#7  2  1
#8  2  2
#```
```


Question:
How to set a cell to NaN in a pandas dataframe I'd like to replace bad values in a column of a dataframe by NaN's.
```
mydata = {'x' : [10, 50, 18, 32, 47, 20], 'y' : ['12', '11', 'N/A', '13', '15', 'N/A']}
df = pd.DataFrame(mydata)

df[df.y == 'N/A']['y'] = np.nan
```
Though, the last line fails and throws a warning because it's working on a copy of df. So, what's the correct way to handle this? I've seen many solutions with iloc or ix but here I need to use a boolean condition.

Response:
just use replace:
```

```python
df.replace('N/A',np.NaN)
#Output
#x    y
#0  10   12
#1  50   11
#2  18  NaN
#3  32   13
#4  47   15
#5  20  NaN
#```
#What you're trying is called chain indexing: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy You can use loc to ensure you operate on the original dF:
#```
```

```python
df.loc[df['y'] == 'N/A','y'] = np.nan
df
#Output
#x    y
#0  10   12
#1  50   11
#2  18  NaN
#3  32   13
#4  47   15
#5  20  NaN
#```
```


Question:
Compare two columns using pandas Using this as a starting point:
```py
a = [['10', '1.2', '4.2'], ['15', '70', '0.03'], ['8', '5', '0']]
df = pd.DataFrame(a, columns=['one', 'two', 'three'])
```
which looks like
```none
one  two three
0   10  1.2   4.2
1   15  70   0.03
2    8   5     0
```
I want to use something like an if statement within pandas.
```py
if df['one'] >= df['two'] and df['one'] <= df['three']:
    df['que'] = df['one']
```
Basically, create a new column by checking each row via the if statement. The docs say to use .all but there is no example...

Response:
You could use np.where. If cond is a boolean array, and A and B are arrays, then
```
C = np.where(cond, A, B)
```
defines C to be equal to A where cond is True, and B where cond is False.
```
import numpy as np
import pandas as pd

a = [['10', '1.2', '4.2'], ['15', '70', '0.03'], ['8', '5', '0']]
df = pd.DataFrame(a, columns=['one', 'two', 'three'])

df['que'] = np.where((df['one'] >= df['two']) & (df['one'] <= df['three'])
                     , df['one'], np.nan)
```
yields
```
one  two three  que
0  10  1.2   4.2   10
1  15   70  0.03  NaN
2   8    5     0  NaN
```
If you have more than one condition, then you could use np.select instead. For example, if you wish df['que'] to equal df['two'] when df['one'] < df['two'], then
```
conditions = [
    (df['one'] >= df['two']) & (df['one'] <= df['three']),
    df['one'] < df['two']]

choices = [df['one'], df['two']]

df['que'] = np.select(conditions, choices, default=np.nan)
```
yields
```
one  two three  que
0  10  1.2   4.2   10
1  15   70  0.03   70
2   8    5     0  NaN
```
If we can assume that df['one'] >= df['two'] when df['one'] < df['two'] is False, then the conditions and choices could be simplified to
```
conditions = [
    df['one'] < df['two'],
    df['one'] <= df['three']]

choices = [df['two'], df['one']]
```
(The assumption may not be true if df['one'] or df['two'] contain NaNs.) Note that
```
a = [['10', '1.2', '4.2'], ['15', '70', '0.03'], ['8', '5', '0']]
df = pd.DataFrame(a, columns=['one', 'two', 'three'])
```
defines a DataFrame with string values. Since they look numeric, you might be better off converting those strings to floats:
```
df2 = df.astype(float)
```
This changes the results, however, since strings compare character-by-character, while floats are compared numerically.
```

```python
'10' <= '4.2'
#Output
#True
```

```python
10 <= 4.2
#Output
#False
#```
```


Question:
Set order of columns in pandas dataframe Is there a way to reorder columns in pandas dataframe based on my personal preference (i.e. not alphabetically or numerically sorted, but more like following certain conventions)? Simple example:
```
frame = pd.DataFrame({
        'one thing':[1,2,3,4],
        'second thing':[0.1,0.2,1,2],
        'other thing':['a','e','i','o']})
```
produces this:
```
one thing other thing  second thing
0          1           a           0.1
1          2           e           0.2
2          3           i           1.0
3          4           o           2.0
```
But instead, I would like this:
```
one thing second thing  other thing
0          1           0.1           a
1          2           0.2           e
2          3           1.0           i
3          4           2.0           o
```
(Please, provide a generic solution rather than specific to this case. Many thanks.)

Response:
Just select the order yourself by typing in the column names. Note the double brackets:
```
frame = frame[['column I want first', 'column I want second'...etc.]]
```


Question:
Convert List to Pandas Dataframe Column I need to convert my list into a one-column pandas dataframe. Current List (len=3):
```none
['Thanks You',
 'Its fine no problem',
 'Are you sure']
```
Required Pandas DF (shape =3,):
```none
0 Thank You
1 Its fine no problem
2 Are you sure
```
N.B. The numbers represent index in the required Pandas DF above.

Response:
Use:
```
L = ['Thanks You', 'Its fine no problem', 'Are you sure']

#create new df
df = pd.DataFrame({'col':L})
print (df)

                   col
0           Thanks You
1  Its fine no problem
2         Are you sure
```
```
df = pd.DataFrame({'oldcol':[1,2,3]})

#add column to existing df
df['col'] = L
print (df)
   oldcol                  col
0       1           Thanks You
1       2  Its fine no problem
2       3         Are you sure
```
Thank you DYZ:
```
#default column name 0
df = pd.DataFrame(L)
print (df)
                     0
0           Thanks You
1  Its fine no problem
2         Are you sure
```