Question:
Pandas, Future Warning: Indexing with multiple keys Pandas throws a Future Warning when I apply a function to multiple columns of a groupby object. It suggests to use a list as index instead of tuples. How would one go about this?
```

```python
df = pd.DataFrame([[1,2,3],[4,5,6],[7,8,9]])
```

```python
df.groupby([0,1])[1,2].apply(sum)
#Output
#<stdin>:1: FutureWarning: Indexing with multiple keys (implicitly converted to a tuple of keys) will be deprecated, use a list instead.
#     1  2
#0 1
#1 2  2  3
#4 5  5  6
#7 8  8  9
#```
```

Response:
This warning was introduced in pandas 1.0.0, following a discussion on GitHub. So best use what was suggested there:
```py
df.groupby([0, 1])[[1, 2]].apply(sum)
```
It's also possible to move the slicing operation to the end, but that is not as efficient:
```py
df.groupby([0, 1]).apply(sum).loc[:, 1:]
```
Thanks @ALollz and @cmosig for helpful comments.


Question:
Sorting by absolute value without changing the data I'm looking for a simple way to sort a pandas dataframe by the absolute value of a particular column, but without actually changing the values within the dataframe. Something similar to sorted(df, key=abs). So if I had a dataframe like:
```
a   b
0   1   -3
1   2   5
2   3   -1
3   4   2
4   5   -9
```
The resultant sorted data when sorting on 'b' would look like:
```
a   b
2   3   -1
3   4   2
0   1   -3
1   2   5
4   5   -9
```

Response:
Perfect Simple Solution with Pandas >= V_1.1.0: Use the parameter key in the sort_values() function:
```
import pandas as pd
df = pd.DataFrame({'A': ['a', 'b', 'c', 'd', 'e', 'f'], 'B': [-3, -2, -1, 0, 1, 2]})

df.sort_values(by='B', key=abs)
```
will yield:
```
A   B
3   d   0
2   c   -1
4   e   1
1   b   -2
5   f   2
0   a   -3
```


Question:
Apply GZIP compression to a CSV in Python Pandas I am trying to write a dataframe to a gzipped csv in python pandas, using the following:
```
import pandas as pd
import datetime
import csv
import gzip

# Get data (with previous connection and script variables)
df = pd.read_sql_query(script, conn)

# Create today's date, to append to file
todaysdatestring = str(datetime.datetime.today().strftime('%Y%m%d'))
print todaysdatestring

# Create csv with gzip compression
df.to_csv('foo-%s.csv.gz' % todaysdatestring,
      sep='|',
      header=True,
      index=False,
      quoting=csv.QUOTE_ALL,
      compression='gzip',
      quotechar='"',
      doublequote=True,
      line_terminator='\n')
```
This just creates a csv called 'foo-YYYYMMDD.csv.gz', not an actual gzip archive. I've also tried adding this:
```
#Turn to_csv statement into a variable
d = df.to_csv('foo-%s.csv.gz' % todaysdatestring,
      sep='|',
      header=True,
      index=False,
      quoting=csv.QUOTE_ALL,
      compression='gzip',
      quotechar='"',
      doublequote=True,
      line_terminator='\n')

# Write above variable to gzip
 with gzip.open('foo-%s.csv.gz' % todaysdatestring, 'wb') as output:
   output.write(d)
```
Which fails as well. Any ideas?

Response: Using df.to_csv() with the keyword argument compression='gzip' should produce a gzip archive. I tested it using same keyword arguments as you, and it worked. You may need to upgrade pandas, as gzip was not implemented until version 0.17.1, but trying to use it on prior versions will not raise an error, and just produce a regular csv. You can determine your current version of pandas by looking at the output of pd.__version__.


Question: Export a LaTeX table from pandas DataFrame Is there an easy way to export a data frame (or even a part of it) to LaTeX? I searched in google and was only able to find solutions using asciitables.

Response:
DataFrames have a to_latex (see the pandas docs) method:
```py

```python
df = pd.DataFrame(np.random.random((5, 5)))
```

```python
df
#Output
#          0         1         2         3         4
#0  0.886864  0.518538  0.359964  0.167291  0.940414
#1  0.834130  0.022920  0.265131  0.059002  0.530584
#2  0.648019  0.953043  0.263551  0.595798  0.153969
#3  0.207003  0.015721  0.931170  0.045044  0.432870
#4  0.039886  0.898780  0.728195  0.112069  0.468485
```

```python
print(df.to_latex())
#Output
#\begin{tabular}{|l|c|c|c|c|c|c|}
#\hline
#{} &         0 &         1 &         2 &         3 &         4 \\
#\hline
#0 &  0.886864 &  0.518538 &  0.359964 &  0.167291 &  0.940414 \\
#1 &  0.834130 &  0.022920 &  0.265131 &  0.059002 &  0.530584 \\
#2 &  0.648019 &  0.953043 &  0.263551 &  0.595798 &  0.153969 \\
#3 &  0.207003 &  0.015721 &  0.931170 &  0.045044 &  0.432870 \\
#4 &  0.039886 &  0.898780 &  0.728195 &  0.112069 &  0.468485 \\
#\hline
#\end{tabular}
#```
#You can simply write this to a tex file. By default latex will render this as: Note: the to_latex (see the pandas docs) method offers several configuration options.
```


Question: Is .ix() always better than .loc() and .iloc() since it is faster and supports integer and label access? I'm learning the Python pandas library. Coming from an R background, the indexing and selecting functions seem more complicated than they need to be. My understanding it that .loc() is only label based and .iloc() is only integer based. Why should I ever use .loc() and .iloc() if .ix() is faster and supports integer and label access?

Response: Please refer to the doc Different Choices for Indexing, it states clearly when and why you should use .loc, .iloc over .ix, it's about explicit use case: .ix supports mixed integer and label based access. It is primarily label based, but will fall back to integer positional access unless the corresponding axis is of integer type. .ix is the most general and will support any of the inputs in .loc and .iloc. .ix also supports floating point label schemes. .ix is exceptionally useful when dealing with mixed positional and label based hierachical indexes. However, when an axis is integer based, ONLY label based access and not positional access is supported. Thus, in such cases, it’s usually better to be explicit and use .iloc or .loc. Update 22 Mar 2017 Thanks to comment from @Alexander, Pandas is going to deprecate ix in 0.20, details in here. One of the strong reason behind is because mixing indexes -- positional and label (effectively using ix) has been a significant source of problems for users. It is expected to migrate to use iloc and loc instead, here is a link on how to convert code.


Question:
Convert pandas DateTimeIndex to Unix Time? What is the idiomatic way of converting a pandas DateTimeIndex to (an iterable of) Unix Time? This is probably not the way to go:
```
[time.mktime(t.timetuple()) for t in my_data_frame.index.to_pydatetime()]
```

Response:
As DatetimeIndex is ndarray under the hood, you can do the conversion without a comprehension (much faster).
```py

```python
import numpy as np
```

```python
import pandas as pd
```

```python
from datetime import datetime
```

```python
dates = [datetime(2012, 5, 1), datetime(2012, 5, 2), datetime(2012, 5, 3)]
   ...: index = pd.DatetimeIndex(dates)
   ...:
```

```python
index.astype(np.int64)
#Output
#array([1335830400000000000, 1335916800000000000, 1336003200000000000],
#        dtype=int64)
```

```python
index.astype(np.int64) // 10**9
#Output
#array([1335830400, 1335916800, 1336003200], dtype=int64)
#
#%timeit [t.value // 10 ** 9 for t in index]
#10000 loops, best of 3: 119 us per loop
#
#%timeit index.astype(np.int64) // 10**9
#100000 loops, best of 3: 18.4 us per loop
#```
```


Question:
Extracting the first day of month of a datetime type column in pandas I have the following dataframe:
```
user_id    purchase_date
  1        2015-01-23 14:05:21
  2        2015-02-05 05:07:30
  3        2015-02-18 17:08:51
  4        2015-03-21 17:07:30
  5        2015-03-11 18:32:56
  6        2015-03-03 11:02:30
```
and purchase_date is a datetime64[ns] column. I need to add a new column df[month] that contains first day of the month of the purchase date:
```
df['month']
2015-01-01
2015-02-01
2015-02-01
2015-03-01
2015-03-01
2015-03-01
```
I'm looking for something like DATE_FORMAT(purchase_date, "%Y-%m-01") m in SQL. I have tried the following code:
```
df['month']=df['purchase_date'].apply(lambda x : x.replace(day=1))
```
It works somehow but returns: 2015-01-01 14:05:21.

Response:
Simpliest and fastest is convert to numpy array by to_numpy and then cast:
```
df['month'] = df['purchase_date'].to_numpy().astype('datetime64[M]')
print (df)
   user_id       purchase_date      month
0        1 2015-01-23 14:05:21 2015-01-01
1        2 2015-02-05 05:07:30 2015-02-01
2        3 2015-02-18 17:08:51 2015-02-01
3        4 2015-03-21 17:07:30 2015-03-01
4        5 2015-03-11 18:32:56 2015-03-01
5        6 2015-03-03 11:02:30 2015-03-01
```
Another solution with floor and pd.offsets.MonthBegin(1) and add pd.offsets.MonthEnd(0) for correct ouput if first day of month:
```
df['month'] = (df['purchase_date'].dt.floor('d') +
                           pd.offsets.MonthEnd(0) - pd.offsets.MonthBegin(1))
print (df)
   user_id       purchase_date      month
0        1 2015-01-23 14:05:21 2015-01-01
1        2 2015-02-05 05:07:30 2015-02-01
2        3 2015-02-18 17:08:51 2015-02-01
3        4 2015-03-21 17:07:30 2015-03-01
4        5 2015-03-11 18:32:56 2015-03-01
5        6 2015-03-03 11:02:30 2015-03-01
```
```
df['month'] = ((df['purchase_date'] + pd.offsets.MonthEnd(0) - pd.offsets.MonthBegin(1))
                         .dt.floor('d'))
print (df)
   user_id       purchase_date      month
0        1 2015-01-23 14:05:21 2015-01-01
1        2 2015-02-05 05:07:30 2015-02-01
2        3 2015-02-18 17:08:51 2015-02-01
3        4 2015-03-21 17:07:30 2015-03-01
4        5 2015-03-11 18:32:56 2015-03-01
5        6 2015-03-03 11:02:30 2015-03-01
```
Last solution is create month period by to_period:
```
df['month'] = df['purchase_date'].dt.to_period('M')
print (df)
   user_id       purchase_date   month
0        1 2015-01-23 14:05:21 2015-01
1        2 2015-02-05 05:07:30 2015-02
2        3 2015-02-18 17:08:51 2015-02
3        4 2015-03-21 17:07:30 2015-03
4        5 2015-03-11 18:32:56 2015-03
5        6 2015-03-03 11:02:30 2015-03
```
... and then to datetimes by to_timestamp, but it is a bit slowier:
```
df['month'] = df['purchase_date'].dt.to_period('M').dt.to_timestamp()
print (df)
   user_id       purchase_date      month
0        1 2015-01-23 14:05:21 2015-01-01
1        2 2015-02-05 05:07:30 2015-02-01
2        3 2015-02-18 17:08:51 2015-02-01
3        4 2015-03-21 17:07:30 2015-03-01
4        5 2015-03-11 18:32:56 2015-03-01
5        6 2015-03-03 11:02:30 2015-03-01
```
There are many solutions, so: Timings (in pandas 1.2.3):
```
rng = pd.date_range('1980-04-01 15:41:12', periods=100000, freq='20H')
df = pd.DataFrame({'purchase_date': rng})
print (df.head())

```python
%timeit df['purchase_date'].to_numpy().astype('datetime64[M]')
8.6 ms ± 27.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

```python
%timeit df['purchase_date'].dt.floor('d') + pd.offsets.MonthEnd(n=0) - pd.offsets.MonthBegin(n=1)
23 ms ± 130 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

```python
%timeit (df['purchase_date'] + pd.offsets.MonthEnd(0) - pd.offsets.MonthBegin(1)).dt.floor('d')
23.6 ms ± 97.9 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

```python
%timeit df['purchase_date'].dt.to_period('M')
9.25 ms ± 215 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

```python
%timeit df['purchase_date'].dt.to_period('M').dt.to_timestamp()
17.6 ms ± 485 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

```python
%timeit df['purchase_date'] + pd.offsets.MonthEnd(0) - pd.offsets.MonthBegin(normalize=True)
23.1 ms ± 116 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

```python
%timeit df['purchase_date'].dt.normalize().map(MonthBegin().rollback)
1.66 s ± 7.16 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
```


Question:
python pandas loc - filter for list of values [duplicate] This question already has answers here: How to filter Pandas dataframe using 'in' and 'not in' like in SQL (12 answers) Closed 7 years ago. This should be incredibly easy, but I can't get it to work. I want to filter my dataset on two or more values.
```
#this works, when I filter for one value
df.loc[df['channel'] == 'sale']

#if I have to filter, two separate columns, I can do this
df.loc[(df['channel'] == 'sale')&(df['type']=='A')]

#but what if I want to filter one column by more than one value?
df.loc[df['channel'] == ('sale','fullprice')]
```
Would this have to be an OR statement? I can do something like in SQL using in?

Response:
There is a df.isin(values) method wich tests whether each element in the DataFrame is contained in values. So, as @MaxU wrote in the comment, you can use
```
df.loc[df['channel'].isin(['sale','fullprice'])]
```
to filter one column by multiple values.


Question:
Convert float64 column to int64 in Pandas I tried to convert a column from data type float64 to int64 using:
```
df['column name'].astype(int64)
```
but got an error: NameError: name 'int64' is not defined The column has number of people but was formatted as 7500000.0, any idea how I can simply change this float64 into int64?

Response:
Solution for pandas 0.24+ for converting numeric with missing values:
```
df = pd.DataFrame({'column name':[7500000.0,7500000.0, np.nan]})
print (df['column name'])
0    7500000.0
1    7500000.0
2          NaN
Name: column name, dtype: float64

df['column name'] = df['column name'].astype(np.int64)
```
ValueError: Cannot convert non-finite values (NA or inf) to integer
```
#http://pandas.pydata.org/pandas-docs/stable/user_guide/integer_na.html
df['column name'] = df['column name'].astype('Int64')
print (df['column name'])
0    7500000
1    7500000
2        NaN
Name: column name, dtype: Int64
```
I think you need cast to numpy.int64:
```
df['column name'].astype(np.int64)
```
Sample:
```
df = pd.DataFrame({'column name':[7500000.0,7500000.0]})
print (df['column name'])
0    7500000.0
1    7500000.0
Name: column name, dtype: float64

df['column name'] = df['column name'].astype(np.int64)
#same as
#df['column name'] = df['column name'].astype(pd.np.int64)
print (df['column name'])
0    7500000
1    7500000
Name: column name, dtype: int64
```
If some NaNs in columns need replace them to some int (e.g. 0) by fillna, because type of NaN is float:
```
df = pd.DataFrame({'column name':[7500000.0,np.nan]})

df['column name'] = df['column name'].fillna(0).astype(np.int64)
print (df['column name'])
0    7500000
1          0
Name: column name, dtype: int64
```
Also check documentation - missing data casting rules EDIT: Convert values with NaNs is buggy:
```
df = pd.DataFrame({'column name':[7500000.0,np.nan]})

df['column name'] = df['column name'].values.astype(np.int64)
print (df['column name'])
0                7500000
1   -9223372036854775808
Name: column name, dtype: int64
```


Question:
Pandas left outer join multiple dataframes on multiple columns I am new to using DataFrame and I would like to know how to perform a SQL equivalent of left outer join on multiple columns on a series of tables Example:
```
df1:
Year    Week    Colour    Val1
2014       A       Red      50
2014       B       Red      60
2014       B     Black      70
2014       C       Red      10
2014       D     Green      20

df2:
Year    Week    Colour    Val2
2014       A     Black      30
2014       B     Black     100
2014       C     Green      50
2014       C       Red      20
2014       D       Red      40

df3:
Year    Week    Colour    Val3
2013       B       Red      60
2013       C     Black      80
2013       B     Black      10
2013       D     Green      20
2013       D       Red      50
```
Essentially I want to do something like this SQL code (Notice that df3 is not joined on Year):
```
SELECT df1.*, df2.Val2, df3.Val3
FROM df1
  LEFT OUTER JOIN df2
    ON df1.Year = df2.Year
    AND df1.Week = df2.Week
    AND df1.Colour = df2.Colour
  LEFT OUTER JOIN df3
    ON df1.Week = df3.Week
    AND df1.Colour = df3.Colour
```
The result should look like:
```
Year    Week    Colour    Val1    Val2    Val3
2014       A       Red      50    Null    Null
2014       B       Red      60    Null      60
2014       B     Black      70     100    Null
2014       C       Red      10      20    Null
2014       D     Green      20    Null    Null
```
I have tried using merge and join but can't figure out how to do it on multiple tables and when there are multiple joints involved. Could someone help me on this please? Thanks

Response:
Merge them in two steps, df1 and df2 first, and then the result of that to df3.
```

```python
s1 = pd.merge(df1, df2, how='left', on=['Year', 'Week', 'Colour'])
```
I dropped year from df3 since you don't need it for the last join.
```
```

```python
df = pd.merge(s1, df3[['Week', 'Colour', 'Val3']],
                       how='left', on=['Week', 'Colour'])
```

```python
df
#Output
#Year Week Colour  Val1  Val2 Val3
#0  2014    A    Red    50   NaN  NaN
#1  2014    B    Red    60   NaN   60
#2  2014    B  Black    70   100   10
#3  2014    C    Red    10    20  NaN
#4  2014    D  Green    20   NaN   20
#
#[5 rows x 6 columns]
#```
```


Question:
error using astype when NaN exists in a dataframe
```
df
     A     B
0   a=10   b=20.10
1   a=20   NaN
2   NaN    b=30.10
3   a=40   b=40.10
```
I tried :
```
df['A'] = df['A'].str.extract('(\d+)').astype(int)
df['B'] = df['B'].str.extract('(\d+)').astype(float)
```
But I get the following error: ValueError: cannot convert float NaN to integer And: AttributeError: Can only use .str accessor with string values, which use np.object_ dtype in pandas How do I fix this ?

Response:
If some values in column are missing (NaN) and then converted to numeric, always dtype is float. You cannot convert values to int. Only to float, because type of NaN is float.
```
print (type(np.nan))
<class 'float'>
```
See docs how convert values if at least one NaN: integer > cast to float64 If need int values you need replace NaN to some int, e.g. 0 by fillna and then it works perfectly:
```
df['A'] = df['A'].str.extract('(\d+)', expand=False)
df['B'] = df['B'].str.extract('(\d+)', expand=False)
print (df)
     A    B
0   10   20
1   20  NaN
2  NaN   30
3   40   40

df1 = df.fillna(0).astype(int)
print (df1)
    A   B
0  10  20
1  20   0
2   0  30
3  40  40

print (df1.dtypes)
A    int32
B    int32
dtype: object
```


Question:
Computing diffs within groups of a dataframe Say I have a dataframe with 3 columns: Date, Ticker, Value (no index, at least to start with). I have many dates and many tickers, but each (ticker, date) tuple is unique. (But obviously the same date will show up in many rows since it will be there for multiple tickers, and the same ticker will show up in multiple rows since it will be there for many dates.) Initially, my rows in a specific order, but not sorted by any of the columns. I would like to compute first differences (daily changes) of each ticker (ordered by date) and put these in a new column in my dataframe. Given this context, I cannot simply do
```
df['diffs'] = df['value'].diff()
```
because adjacent rows do not come from the same ticker. Sorting like this:
```
df = df.sort(['ticker', 'date'])
df['diffs'] = df['value'].diff()
```
doesn't solve the problem because there will be "borders". I.e. after that sort, the last value for one ticker will be above the first value for the next ticker. And computing differences then would take a difference between two tickers. I don't want this. I want the earliest date for each ticker to wind up with an NaN in its diff column. This seems like an obvious time to use groupby but for whatever reason, I can't seem to get it to work properly. To be clear, I would like to perform the following process: Group rows based on their ticker Within each group, sort rows by their date Within each sorted group, compute differences of the value column Put these differences into the original dataframe in a new diffs column (ideally leaving the original dataframe order in tact.) I have to imagine this is a one-liner. But what am I missing? Edit at 9:00pm 2013-12-17 Ok...some progress. I can do the following to get a new dataframe:
```
result = df.set_index(['ticker', 'date'])\
    .groupby(level='ticker')\
    .transform(lambda x: x.sort_index().diff())\
    .reset_index()
```
But if I understand the mechanics of groupby, my rows will now be sorted first by ticker and then by date. Is that correct? If so, would I need to do a merge to append the differences column (currently in result['current'] to the original dataframe df?

Response:
wouldn't be just easier to do what yourself describe, namely
```
df.sort(['ticker', 'date'], inplace=True)
df['diffs'] = df['value'].diff()
```
and then correct for borders:
```
mask = df.ticker != df.ticker.shift(1)
df['diffs'][mask] = np.nan
```
to maintain the original index you may do idx = df.index in the beginning, and then at the end you can do df.reindex(idx), or if it is a huge dataframe, perform the operations on
```
df.filter(['ticker', 'date', 'value'])
```
and then join the two dataframes at the end. edit: alternatively, ( though still not using groupby )
```
df.set_index(['ticker','date'], inplace=True)
df.sort_index(inplace=True)
df['diffs'] = np.nan

for idx in df.index.levels[0]:
    df.diffs[idx] = df.value[idx].diff()
```
for
```
date ticker  value
0    63      C   1.65
1    88      C  -1.93
2    22      C  -1.29
3    76      A  -0.79
4    72      B  -1.24
5    34      A  -0.23
6    92      B   2.43
7    22      A   0.55
8    32      A  -2.50
9    59      B  -1.01
```
this will produce:
```
value  diffs
ticker date
A      22     0.55    NaN
       32    -2.50  -3.05
       34    -0.23   2.27
       76    -0.79  -0.56
B      59    -1.01    NaN
       72    -1.24  -0.23
       92     2.43   3.67
C      22    -1.29    NaN
       63     1.65   2.94
       88    -1.93  -3.58
```


Question:
How to read a JSON file with nested objects as a pandas DataFrame? I am curious how I can use pandas to read nested json of the following structure:
```
{
    "number": "",
    "date": "01.10.2016",
    "name": "R 3932",
    "locations": [
        {
            "depTimeDiffMin": "0",
            "name": "Spital am Pyhrn Bahnhof",
            "arrTime": "",
            "depTime": "06:32",
            "platform": "2",
            "stationIdx": "0",
            "arrTimeDiffMin": "",
            "track": "R 3932"
        },
        {
            "depTimeDiffMin": "0",
            "name": "Windischgarsten Bahnhof",
            "arrTime": "06:37",
            "depTime": "06:40",
            "platform": "2",
            "stationIdx": "1",
            "arrTimeDiffMin": "1",
            "track": ""
        },
        {
            "depTimeDiffMin": "",
            "name": "Linz/Donau Hbf",
            "arrTime": "08:24",
            "depTime": "",
            "platform": "1A-B",
            "stationIdx": "22",
            "arrTimeDiffMin": "1",
            "track": ""
        }
    ]
}
```
This here keeps the array as json. I would rather prefer it to be expanded into columns.
```
pd.read_json("/myJson.json", orient='records')
```
edit Thanks for the first answers. I should refine my question: A flattening of the nested attributes in the array is not mandatory. It would be ok to just [A, B, C] concatenate the df.locations['name']. My file contains multiple JSON objects (1 per line) I would like to keep number, date, name, and locations column. However, I would need to join the locations.
```
allLocations = ""
isFirst = True
for location in result.locations:
    if isFirst:
        isFirst = False
        allLocations = location['name']
    else:
        allLocations += "; " + location['name']
allLocations
```
My approach here does not seem to be efficient / pandas style.

Response:
You can use json_normalize:
```
import json

with open('myJson.json') as data_file:
    data = json.load(data_file)

df = pd.json_normalize(data, 'locations', ['date', 'number', 'name'],
                    record_prefix='locations_')
print (df)
  locations_arrTime locations_arrTimeDiffMin locations_depTime  \
0                                                        06:32
1             06:37                        1             06:40
2             08:24                        1

  locations_depTimeDiffMin           locations_name locations_platform  \
0                        0  Spital am Pyhrn Bahnhof                  2
1                        0  Windischgarsten Bahnhof                  2
2                                    Linz/Donau Hbf               1A-B

  locations_stationIdx locations_track number    name        date
0                    0          R 3932         R 3932  01.10.2016
1                    1                         R 3932  01.10.2016
2                   22                         R 3932  01.10.2016
```
EDIT: You can use read_json with parsing name by DataFrame constructor and last groupby with apply join:
```
df = pd.read_json("myJson.json")
df.locations = pd.DataFrame(df.locations.values.tolist())['name']
df = df.groupby(['date','name','number'])['locations'].apply(','.join).reset_index()
print (df)
        date    name number                                          locations
0 2016-01-10  R 3932         Spital am Pyhrn Bahnhof,Windischgarsten Bahnho...
```


Question:
Applying uppercase to a column in pandas dataframe I'm having trouble applying upper case to a column in my DataFrame. dataframe is df. 1/2 ID is the column head that need to apply UPPERCASE. The problem is that the values are made up of three letters and three numbers. For example rrr123 is one of the values.
```
df['1/2 ID'] = map(str.upper, df['1/2 ID'])
```
I got an error: TypeError: descriptor 'upper' requires a 'str' object but received a 'unicode' error. How can I apply upper case to the first three letters in the column of the DataFrame df?

Response:
If your version of pandas is a recent version then you can just use the vectorised string method upper:
```
df['1/2 ID'] = df['1/2 ID'].str.upper()
```
This method does not work inplace, so the result must be assigned back.


Question:
How to add an extra row to a pandas dataframe [duplicate] This question already has answers here: Create a Pandas Dataframe by appending one row at a time [duplicate] (32 answers) Closed 8 years ago. If I have an empty dataframe as such:
```
columns = ['Date', 'Name', 'Action','ID']
df = pd.DataFrame(columns=columns)
```
Is there a way to append a new row to this newly created dataframe? Currently I have to create a dictionary, populate it, then append the dictionary to the dataframe at the end. Is there a more direct way?

Response:
Try this:
```
df.loc[len(df)] = ['8/19/2014', 'Jun', 'Fly', '98765']
```
Warning: this method works only if there are no "holes" in the index. For example, suppose you have a DataFrame with three rows, with indices 0, 1, and 3 (for example, because you deleted row number 2). Then, len(df) = 3, so by the above command does not add a new row - it overrides row number 3.


Question: what's the inverse of the quantile function on a pandas Series? The quantile functions gives us the quantile of a given pandas series s, E.g. s.quantile(0.9) is 4.2 Is there the inverse function (i.e. cumulative distribution) which finds the value x such that s.quantile(x)=4 Thanks

Response:
Use scipy.stats.percentileofscore:
```
# libs required
from scipy import stats
import pandas as pd
import numpy as np

# generate ramdom data with same seed (to be reproducible)
np.random.seed(seed=1)
df = pd.DataFrame(np.random.uniform(0, 1, (10)), columns=['a'])

# quantile function
x = df.quantile(0.5)[0]

# inverse of quantile
stats.percentileofscore(df['a'], x)
```


Question:
Boxplot of Multiple Columns of a Pandas Dataframe on the Same Figure (seaborn) I feel I am probably not thinking of something obvious. I want to put in the same figure, the box plot of every column of a dataframe, where on the x-axis I have the columns' names. In the seaborn.boxplot() this would be equal to groupby by every column. In pandas I would do
```
df = pd.DataFrame(data = np.random.random(size=(4,4)), columns = ['A','B','C','D'])
df.boxplot()
```
which yields Now I would like to get the same thing in seaborn. But when I try sns.boxplot(df), I get only one grouped boxplot. How do I reproduce the same figure in seaborn?

Response:
The seaborn equivalent of
```
df.boxplot()
```
is
```
sns.boxplot(x="variable", y="value", data=pd.melt(df))
```
or just
```
sns.boxplot(data=df)
```
which will plot any column of numeric values, without converting the DataFrame from a wide to long format, using seaborn v0.11.1. This will create a single figure, with a separate boxplot for each column. Complete example with melt:
```
import numpy as np; np.random.seed(42)
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

df = pd.DataFrame(data = np.random.random(size=(4,4)), columns = ['A','B','C','D'])

sns.boxplot(x="variable", y="value", data=pd.melt(df))

plt.show()
```
This works because pd.melt converts a wide-form dataframe
```
A         B         C         D
0  0.374540  0.950714  0.731994  0.598658
1  0.156019  0.155995  0.058084  0.866176
2  0.601115  0.708073  0.020584  0.969910
3  0.832443  0.212339  0.181825  0.183405
```
to long-form
```
variable     value
0         A  0.374540
1         A  0.156019
2         A  0.601115
3         A  0.832443
4         B  0.950714
5         B  0.155995
6         B  0.708073
7         B  0.212339
8         C  0.731994
9         C  0.058084
10        C  0.020584
11        C  0.181825
12        D  0.598658
13        D  0.866176
14        D  0.969910
15        D  0.183405
```


Question:
Merge a list of dataframes to create one dataframe [duplicate] This question already has answers here: Pandas Merging 101 (8 answers) Closed 6 years ago. I have a list of 18 data frames:
```
dfList = [df1, df2, df3, df4, df5, df6.....df18]
```
All of the data frames have a common id column so it's easy to join them each together with pd.merge 2 at a time. Is there a way to join them all at once so that dfList comes back as a single dataframe?

Response:
I think you need concat, but first set index of each DataFrame by common column:
```
dfs = [df.set_index('id') for df in dfList]
print pd.concat(dfs, axis=1)
```
If need join by merge:
```
from functools import reduce
df = reduce(lambda df1,df2: pd.merge(df1,df2,on='id'), dfList)
```


Question:
Python: Random selection per group Say that I have a dataframe that looks like:
```
Name Group_Id
AAA  1
ABC  1
CCC  2
XYZ  2
DEF  3
YYH  3
```
How could I randomly select one (or more) row for each Group_Id? Say that I want one random draw per Group_Id, I would get:
```
Name Group_Id
AAA  1
XYZ  2
DEF  3
```

Response:
From 0.16.x onwards pd.DataFrame.sample provides a way to return a random sample of items from an axis of object.
```

```python
df.groupby('Group_Id').apply(lambda x: x.sample(1)).reset_index(drop=True)
#Output
#Name  Group_Id
#0  ABC         1
#1  XYZ         2
#2  DEF         3
#```
```


Question: forward fill specific columns in pandas dataframe If I have a dataframe with multiple columns ['x', 'y', 'z'], how do I forward fill only one column 'x'? Or a group of columns ['x','y']? I only know how to do it by axis.

Response:
tl;dr:
```
cols = ['X', 'Y']
df.loc[:,cols] = df.loc[:,cols].ffill()
```
And I have also added a self containing example:
```

```python
import pandas as pd
```

```python
import numpy as np
```

```python

```

```python
## create dataframe
ts1 = [0, 1, np.nan, np.nan, np.nan, np.nan]
```

```python
ts2 = [0, 2, np.nan, 3, np.nan, np.nan]
```

```python
d =  {'X': ts1, 'Y': ts2, 'Z': ts2}
```

```python
df = pd.DataFrame(data=d)
```

```python
print(df.head())
#Output
#    X   Y   Z
#0   0   0   0
#1   1   2   2
#2 NaN NaN NaN
#3 NaN   3   3
#4 NaN NaN NaN
```

```python

```

```python
## apply forward fill
cols = ['X', 'Y']
```

```python
df.loc[:,cols] = df.loc[:,cols].ffill()
```

```python
print(df.head())
#Output
#   X  Y   Z
#0  0  0   0
#1  1  2   2
#2  1  2 NaN
#3  1  3   3
#4  1  3 NaN
#```
```


Question:
pandas: how to run a pivot with a multi-index? I would like to run a pivot on a pandas DataFrame, with the index being two columns, not one. For example, one field for the year, one for the month, an 'item' field which shows 'item 1' and 'item 2' and a 'value' field with numerical values. I want the index to be year + month. The only way I managed to get this to work was to combine the two fields into one, then separate them again. is there a better way? Minimal code copied below. Thanks a lot! PS Yes, I am aware there are other questions with the keywords 'pivot' and 'multi-index', but I did not understand if/how they can help me with this question.
```
import pandas as pd
import numpy as np

df= pd.DataFrame()
month = np.arange(1, 13)
values1 = np.random.randint(0, 100, 12)
values2 = np.random.randint(200, 300, 12)

df['month'] = np.hstack((month, month))
df['year'] = 2004
df['value'] = np.hstack((values1, values2))
df['item'] = np.hstack((np.repeat('item 1', 12), np.repeat('item 2', 12)))

# This doesn't work:
# ValueError: Wrong number of items passed 24, placement implies 2
# mypiv = df.pivot(['year', 'month'], 'item', 'value')

# This doesn't work, either:
# df.set_index(['year', 'month'], inplace=True)
# ValueError: cannot label index with a null key
# mypiv = df.pivot(columns='item', values='value')

# This below works but is not ideal:
# I have to first concatenate then separate the fields I need
df['new field'] = df['year'] * 100 + df['month']

mypiv = df.pivot('new field', 'item', 'value').reset_index()
mypiv['year'] = mypiv['new field'].apply( lambda x: int(x) / 100)
mypiv['month'] = mypiv['new field'] % 100
```

Response:
You can group and then unstack.
```

```python
df.groupby(['year', 'month', 'item'])['value'].sum().unstack('item')
#Output
#item        item 1  item 2
#year month
#2004 1          33     250
#     2          44     224
#     3          41     268
#     4          29     232
#     5          57     252
#     6          61     255
#     7          28     254
#     8          15     229
#     9          29     258
#     10         49     207
#     11         36     254
#     12         23     209
#```
#Or use pivot_table:
#```
```

```python
df.pivot_table(
#Output
#        values='value',
#        index=['year', 'month'],
#        columns='item',
#        aggfunc=np.sum)
#item        item 1  item 2
#year month
#2004 1          33     250
#     2          44     224
#     3          41     268
#     4          29     232
#     5          57     252
#     6          61     255
#     7          28     254
#     8          15     229
#     9          29     258
#     10         49     207
#     11         36     254
#     12         23     209
#```
```


Question:
summing two columns in a pandas dataframe when I use this syntax it creates a series rather than adding a column to my new dataframe sum. My code:
```
sum = data['variance'] = data.budget + data.actual
```
My dataframe data currently has everything except the budget - actual column. How do I create a variance column?
```
cluster  date                  budget  actual budget - actual
0   a        2014-01-01  00:00:00  11000   10000       1000
1   a        2014-02-01  00:00:00  1200    1000
2   a        2014-03-01  00:00:00  200     100
3   b        2014-04-01  00:00:00  200     300
4   b        2014-05-01  00:00:00  400     450
5   c        2014-06-01  00:00:00  700     1000
6   c        2014-07-01  00:00:00  1200    1000
7   c        2014-08-01  00:00:00  200     100
8   c        2014-09-01  00:00:00  200     300
```

Response:
I think you've misunderstood some python syntax, the following does two assignments:
```

```python
a = b = 1
```

```python
a
#Output
#1
```

```python
b
#Output
#1
#```
#So in your code it was as if you were doing:
#```
#sum = df['budget'] + df['actual']  # a Series
## and
#df['variance'] = df['budget'] + df['actual']  # assigned to a column
#```
#The latter creates a new column for df:
#```
```

```python
df
#Output
#cluster                 date  budget  actual
#0       a  2014-01-01 00:00:00   11000   10000
#1       a  2014-02-01 00:00:00    1200    1000
#2       a  2014-03-01 00:00:00     200     100
#3       b  2014-04-01 00:00:00     200     300
#4       b  2014-05-01 00:00:00     400     450
#5       c  2014-06-01 00:00:00     700    1000
#6       c  2014-07-01 00:00:00    1200    1000
#7       c  2014-08-01 00:00:00     200     100
#8       c  2014-09-01 00:00:00     200     300
```

```python
df['variance'] = df['budget'] + df['actual']
```

```python
df
#Output
#cluster                 date  budget  actual  variance
#0       a  2014-01-01 00:00:00   11000   10000     21000
#1       a  2014-02-01 00:00:00    1200    1000      2200
#2       a  2014-03-01 00:00:00     200     100       300
#3       b  2014-04-01 00:00:00     200     300       500
#4       b  2014-05-01 00:00:00     400     450       850
#5       c  2014-06-01 00:00:00     700    1000      1700
#6       c  2014-07-01 00:00:00    1200    1000      2200
#7       c  2014-08-01 00:00:00     200     100       300
#8       c  2014-09-01 00:00:00     200     300       500
#```
#As an aside, you shouldn't use sum as a variable name as the overrides the built-in sum function.
```


Question:
pandas - Merge nearly duplicate rows based on column value I have a pandas dataframe with several rows that are near duplicates of each other, except for one value. My goal is to merge or "coalesce" these rows into a single row, without summing the numerical values. Here is an example of what I'm working with:
```
Name   Sid   Use_Case  Revenue
A      xx01  Voice     $10.00
A      xx01  SMS       $10.00
B      xx02  Voice     $5.00
C      xx03  Voice     $15.00
C      xx03  SMS       $15.00
C      xx03  Video     $15.00
```
And here is what I would like:
```
Name   Sid   Use_Case            Revenue
A      xx01  Voice, SMS          $10.00
B      xx02  Voice               $5.00
C      xx03  Voice, SMS, Video   $15.00
```
The reason I don't want to sum the "Revenue" column is because my table is the result of doing a pivot over several time periods where "Revenue" simply ends up getting listed multiple times instead of having a different value per "Use_Case". What would be the best way to tackle this issue? I've looked into the groupby() function but I still don't understand it very well.

Response:
I think you can use groupby with aggregate first and custom function ', '.join:
```
df = df.groupby('Name').agg({'Sid':'first',
                             'Use_Case': ', '.join,
                             'Revenue':'first' }).reset_index()

#change column order
print df[['Name','Sid','Use_Case','Revenue']]
  Name   Sid           Use_Case Revenue
0    A  xx01         Voice, SMS  $10.00
1    B  xx02              Voice   $5.00
2    C  xx03  Voice, SMS, Video  $15.00
```
Nice idea from comment, thanks Goyo:
```
df = df.groupby(['Name','Sid','Revenue'])['Use_Case'].apply(', '.join).reset_index()

#change column order
print df[['Name','Sid','Use_Case','Revenue']]
  Name   Sid           Use_Case Revenue
0    A  xx01         Voice, SMS  $10.00
1    B  xx02              Voice   $5.00
2    C  xx03  Voice, SMS, Video  $15.00
```


Question:
pandas read_json: "If using all scalar values, you must pass an index" I have some difficulty in importing a JSON file with pandas.
```
import pandas as pd
map_index_to_word = pd.read_json('people_wiki_map_index_to_word.json')
```
This is the error that I get:
```
ValueError: If using all scalar values, you must pass an index
```
The file structure is simplified like this:
```
{"biennials": 522004, "lb915": 116290, "shatzky": 127647, "woode": 174106, "damfunk": 133206, "nualart": 153444, "hatefillot": 164111, "missionborn": 261765, "yeardescribed": 161075, "theoryhe": 521685}
```
It is from the machine learning course of University of Washington on Coursera. You can find the file here.

Response:
Try
```
ser = pd.read_json('people_wiki_map_index_to_word.json', typ='series')
```
That file only contains key value pairs where values are scalars. You can convert it to a dataframe with ser.to_frame('count'). You can also do something like this:
```
import json
with open('people_wiki_map_index_to_word.json', 'r') as f:
    data = json.load(f)
```
Now data is a dictionary. You can pass it to a dataframe constructor like this:
```
df = pd.DataFrame({'count': data})
```


Question:
Repeat Rows in Data Frame n Times [duplicate] This question already has answers here: Replicating rows in a pandas data frame by a column value [duplicate] (4 answers) Closed 7 years ago. consider a data frame defined like so:
```
import pandas as pd
test = pd.DataFrame({
    'id' : ['a', 'b', 'c', 'd'],
    'times' : [2, 3, 1, 5]
})
```
Is it possible to create a new data frame from this in which each row is repeated times times, such that the result looks like this:
```

```python
result
#Output
#   id  times
#0   a      2
#1   a      2
#2   b      3
#3   b      3
#4   b      3
#5   c      1
#6   d      5
#7   d      5
#8   d      5
#9   d      5
#10  d      5
#```
```

Response:
Use a combination of pd.DataFrame.loc and pd.Index.repeat
```
test.loc[test.index.repeat(test.times)]

  id  times
0  a      2
0  a      2
1  b      3
1  b      3
1  b      3
2  c      1
3  d      5
3  d      5
3  d      5
3  d      5
3  d      5
```
To mimic your exact output, use reset_index
```
test.loc[test.index.repeat(test.times)].reset_index(drop=True)

   id  times
0   a      2
1   a      2
2   b      3
3   b      3
4   b      3
5   c      1
6   d      5
7   d      5
8   d      5
9   d      5
10  d      5
```


Question:
Ranking order per group in Pandas Consider a dataframe with three columns: group_ID, item_ID and value. Say we have 10 itemIDs total. I need to rank each item_ID (1 to 10) within each group_ID based on value , and then see the mean rank (and other stats) across groups (e.g. the IDs with the highest value across groups would get ranks closer to 1). How can I do this in Pandas? This answer does something very close with qcut, but not exactly the same. A data example would look like:
```none
group_ID   item_ID  value
0   0S00A1HZEy        AB     10
1   0S00A1HZEy        AY      4
2   0S00A1HZEy        AC     35
3   0S03jpFRaC        AY     90
4   0S03jpFRaC        A5      3
5   0S03jpFRaC        A3     10
6   0S03jpFRaC        A2      8
7   0S03jpFRaC        A4      9
8   0S03jpFRaC        A6      2
9   0S03jpFRaC        AX      0
```
which would result in:
```none
group_ID   item_ID   rank
0   0S00A1HZEy        AB      2
1   0S00A1HZEy        AY      3
2   0S00A1HZEy        AC      1
3   0S03jpFRaC        AY      1
4   0S03jpFRaC        A5      5
5   0S03jpFRaC        A3      2
6   0S03jpFRaC        A2      4
7   0S03jpFRaC        A4      3
8   0S03jpFRaC        A6      6
9   0S03jpFRaC        AX      7
```

Response:
There are lots of different arguments you can pass to rank; it looks like you can use rank("dense", ascending=False) to get the results you want, after doing a groupby:
```

```python
df["rank"] = df.groupby("group_ID")["value"].rank(method="dense", ascending=False)
```

```python
df
#Output
#     group_ID item_ID  value  rank
#0  0S00A1HZEy      AB     10     2
#1  0S00A1HZEy      AY      4     3
#2  0S00A1HZEy      AC     35     1
#3  0S03jpFRaS      AY     90     1
#4  0S03jpFRaS      A5      3     5
#5  0S03jpFRaS      A3     10     2
#6  0S03jpFRaS      A2      8     4
#7  0S03jpFRaS      A4      9     3
#8  0S03jpFRaS      A6      2     6
#9  0S03jpFRaS      AX      0     7
#```
#But note that if you're not using a global ranking scheme, finding out the mean rank across groups isn't very meaningful-- unless there are duplicate values in a group (and so you have duplicate rank values) all you're doing is measuring how many elements there are in a group.
```


Question:
Tilde sign in pandas DataFrame I'm new to python/pandas and came across a code snippet.
```
df = df[~df['InvoiceNo'].str.contains('C')]
```
Would be much obliged if I could know what is the tilde sign's usage in this context?

Response:
It means bitwise not, inversing boolean mask - Falses to Trues and Trues to Falses. Sample:
```
df = pd.DataFrame({'InvoiceNo': ['aaC','ff','lC'],
                   'a':[1,2,5]})
print (df)
  InvoiceNo  a
0       aaC  1
1        ff  2
2        lC  5

#check if column contains C
print (df['InvoiceNo'].str.contains('C'))
0     True
1    False
2     True
Name: InvoiceNo, dtype: bool

#inversing mask
print (~df['InvoiceNo'].str.contains('C'))
0    False
1     True
2    False
Name: InvoiceNo, dtype: bool
```
Filter by boolean indexing:
```
df = df[~df['InvoiceNo'].str.contains('C')]
print (df)
  InvoiceNo  a
1        ff  2
```
So output is all rows of DataFrame, which not contains C in column InvoiceNo.


Question:
Pandas: Sampling a DataFrame [duplicate] This question already has answers here: How to split data into 3 sets (train, validation and test)? (13 answers) Closed 8 years ago. I'm trying to read a fairly large CSV file with Pandas and split it up into two random chunks, one of which being 10% of the data and the other being 90%. Here's my current attempt:
```
rows = data.index
row_count = len(rows)
random.shuffle(list(rows))

data.reindex(rows)

training_data = data[row_count // 10:]
testing_data = data[:row_count // 10]
```
For some reason, sklearn throws this error when I try to use one of these resulting DataFrame objects inside of a SVM classifier:
```
IndexError: each subindex must be either a slice, an integer, Ellipsis, or newaxis
```
I think I'm doing it wrong. Is there a better way to do this?

Response:
What version of pandas are you using? For me your code works fine (i`m on git master). Another approach could be:
```

```python
import pandas
```

```python
import random
```

```python
df = pandas.DataFrame(np.random.randn(100, 4), columns=list('ABCD'))
```

```python
rows = random.sample(df.index, 10)
```

```python
df_10 = df.ix[rows]
```

```python
df_90 = df.drop(rows)
```
Newer version (from 0.16.1 on) supports this directly: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sample.html
```


Question:
Fillna in multiple columns in place in Python Pandas I have a pandas dataFrame of mixed types, some are strings and some are numbers. I would like to replace the NAN values in string columns by '.', and the NAN values in float columns by 0. Consider this small fictitious example:
```
df = pd.DataFrame({'Name':['Jack','Sue',pd.np.nan,'Bob','Alice','John'],
    'A': [1, 2.1, pd.np.nan, 4.7, 5.6, 6.8],
    'B': [.25, pd.np.nan, pd.np.nan, 4, 12.2, 14.4],
    'City':['Seattle','SF','LA','OC',pd.np.nan,pd.np.nan]})
```
Now, I can do it in 3 lines:
```
df['Name'].fillna('.',inplace=True)
df['City'].fillna('.',inplace=True)
df.fillna(0,inplace=True)
```
Since this is a small dataframe, 3 lines is probably ok. In my real example (which I cannot share here due to data confidentiality reasons), I have many more string columns and numeric columns. SO I end up writing many lines just for fillna. Is there a concise way of doing this?

Response:
Came across this page while looking for an answer to this problem, but didn't like the existing answers. I ended up finding something better in the DataFrame.fillna documentation, and figured I'd contribute for anyone else that happens upon this. If you have multiple columns, but only want to replace the NaN in a subset of them, you can use:
```
df.fillna({'Name':'.', 'City':'.'}, inplace=True)
```
This also allows you to specify different replacements for each column. And if you want to go ahead and fill all remaining NaN values, you can just throw another fillna on the end:
```
df.fillna({'Name':'.', 'City':'.'}, inplace=True).fillna(0, inplace=True)
```
Edit (22 Apr 2021) Functionality (presumably / apparently) changed since original post, and you can no longer chain 2 inplace fillna() operations. You can still chain, but now must assign that chain to the df instead of modifying in place, e.g. like so:
```
df = df.fillna({'Name':'.', 'City':'.'}).fillna(0)
```


Question:
Python remove stop words from pandas dataframe I want to remove the stop words from my column "tweets". How do I iterative over each row and each item?
```
pos_tweets = [('I love this car', 'positive'),
    ('This view is amazing', 'positive'),
    ('I feel great this morning', 'positive'),
    ('I am so excited about the concert', 'positive'),
    ('He is my best friend', 'positive')]

test = pd.DataFrame(pos_tweets)
test.columns = ["tweet","class"]
test["tweet"] = test["tweet"].str.lower().str.split()

from nltk.corpus import stopwords
stop = stopwords.words('english')
```

Response:
We can import stopwords from nltk.corpus as below. With that, We exclude stopwords with Python's list comprehension and pandas.DataFrame.apply.
```py
# Import stopwords with nltk.
from nltk.corpus import stopwords
stop = stopwords.words('english')

pos_tweets = [('I love this car', 'positive'),
    ('This view is amazing', 'positive'),
    ('I feel great this morning', 'positive'),
    ('I am so excited about the concert', 'positive'),
    ('He is my best friend', 'positive')]

test = pd.DataFrame(pos_tweets)
test.columns = ["tweet","class"]

# Exclude stopwords with Python's list comprehension and pandas.DataFrame.apply.
test['tweet_without_stopwords'] = test['tweet'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))
print(test)
# Out[40]:
#                                tweet     class tweet_without_stopwords
# 0                    I love this car  positive              I love car
# 1               This view is amazing  positive       This view amazing
# 2          I feel great this morning  positive    I feel great morning
# 3  I am so excited about the concert  positive       I excited concert
# 4               He is my best friend  positive          He best friend
```
It can also be excluded by using pandas.Series.str.replace.
```py
pat = r'\b(?:{})\b'.format('|'.join(stop))
test['tweet_without_stopwords'] = test['tweet'].str.replace(pat, '')
test['tweet_without_stopwords'] = test['tweet_without_stopwords'].str.replace(r'\s+', ' ')
# Same results.
# 0              I love car
# 1       This view amazing
# 2    I feel great morning
# 3       I excited concert
# 4          He best friend
```
If you can not import stopwords, you can download as follows.
```py
import nltk
nltk.download('stopwords')
```
Another way to answer is to import text.ENGLISH_STOP_WORDS from sklearn.feature_extraction.
```py
# Import stopwords with scikit-learn
from sklearn.feature_extraction import text
stop = text.ENGLISH_STOP_WORDS
```
Notice that the number of words in the scikit-learn stopwords and nltk stopwords are different.


Question:
Pandas convert string to int I have a large dataframe with ID numbers:
```
ID.head()
Out[64]:
0    4806105017087
1    4806105017087
2    4806105017087
3    4901295030089
4    4901295030089
```
These are all strings at the moment. I want to convert to int without using loops - for this I use ID.astype(int). The problem is that some of my lines contain dirty data which cannot be converted to int, for e.g.
```
ID[154382]
Out[58]: 'CN414149'
```
How can I (without using loops) remove these type of occurrences so that I can use astype with peace of mind?

Response:
You need add parameter errors='coerce' to function to_numeric:
```
ID = pd.to_numeric(ID, errors='coerce')
```
If ID is column:
```
df.ID = pd.to_numeric(df.ID, errors='coerce')
```
but non numeric are converted to NaN, so all values are float. For int need convert NaN to some value e.g. 0 and then cast to int:
```
df.ID = pd.to_numeric(df.ID, errors='coerce').fillna(0).astype(np.int64)
```
Sample:
```
df = pd.DataFrame({'ID':['4806105017087','4806105017087','CN414149']})
print (df)
              ID
0  4806105017087
1  4806105017087
2       CN414149

print (pd.to_numeric(df.ID, errors='coerce'))
0    4.806105e+12
1    4.806105e+12
2             NaN
Name: ID, dtype: float64

df.ID = pd.to_numeric(df.ID, errors='coerce').fillna(0).astype(np.int64)
print (df)
              ID
0  4806105017087
1  4806105017087
2              0
```
EDIT: If use pandas 0.25+ then is possible use integer_na:
```
df.ID = pd.to_numeric(df.ID, errors='coerce').astype('Int64')
print (df)
              ID
0  4806105017087
1  4806105017087
2            NaN
```


Question:
How to pivot a dataframe in Pandas? I have a table in csv format that looks like this. I would like to transpose the table so that the values in the indicator column are the new columns.
```
Indicator  Country  Year  Value
1          Angola   2005  6
2          Angola   2005  13
3          Angola   2005  10
4          Angola   2005  11
5          Angola   2005  5
1          Angola   2006  3
2          Angola   2006  2
3          Angola   2006  7
4          Angola   2006  3
5          Angola   2006  6
```
I would like the end result to look like this:
```
Country  Year  1  2   3   4   5
Angola   2005  6  13  10  11  5
Angola   2006  3  2   7   3   6
```
I have tried using a pandas data frame with not much success.
```
print(df.pivot(columns = 'Country', 'Year', 'Indicator', values = 'Value'))
```
Any thoughts on how to accomplish this?

Response:
You can use .pivot like so:
```
out = df.pivot(index=['Country', 'Year'], columns='Indicator', values='Value')
print(out)
```
```
Indicator     1   2   3   4  5
Country Year
Angola  2005  6  13  10  11  5
        2006  3   2   7   3  6
```
To squash back into a flat table, use .rename_axis to remove "Indicator" and .reset_index to convert Country and Year back to normal columns.
```
print(out.rename_axis(columns=None).reset_index())
```
```
Country  Year  1   2   3   4  5
0  Angola  2005  6  13  10  11  5
1  Angola  2006  3   2   7   3  6
```
.pivot_table If your full data has duplicate combinations of labels (Country, Year, Indicator), you can use .pivot_table. It takes the mean by default.
```
out = df.pivot_table(
    index=['Country', 'Year'],
    columns='Indicator',
    values='Value')
print(out.rename_axis(columns=None).reset_index())
```
```
Country  Year    1     2     3     4    5
0  Angola  2005  6.0  13.0  10.0  11.0  5.0
1  Angola  2006  3.0   2.0   7.0   3.0  6.0
```
User guide: Reshaping and pivot tables


Question:
Difference between pivot and pivot_table. Why is only pivot_table working? I have the following dataframe.
```
df.head(30)

     struct_id  resNum score_type_name  score_value
0   4294967297       1           omega     0.064840
1   4294967297       1          fa_dun     2.185618
2   4294967297       1      fa_dun_dev     0.000027
3   4294967297       1     fa_dun_semi     2.185591
4   4294967297       1             ref    -1.191180
5   4294967297       2            rama    -0.795161
6   4294967297       2           omega     0.222345
7   4294967297       2          fa_dun     1.378923
8   4294967297       2      fa_dun_dev     0.028560
9   4294967297       2      fa_dun_rot     1.350362
10  4294967297       2         p_aa_pp    -0.442467
11  4294967297       2             ref     0.249477
12  4294967297       3            rama     0.267443
13  4294967297       3           omega     0.005106
14  4294967297       3          fa_dun     0.020352
15  4294967297       3      fa_dun_dev     0.025507
16  4294967297       3      fa_dun_rot    -0.005156
17  4294967297       3         p_aa_pp    -0.096847
18  4294967297       3             ref     0.979644
19  4294967297       4            rama    -1.403292
20  4294967297       4           omega     0.212160
21  4294967297       4          fa_dun     4.218029
22  4294967297       4      fa_dun_dev     0.003712
23  4294967297       4     fa_dun_semi     4.214317
24  4294967297       4         p_aa_pp    -0.462765
25  4294967297       4             ref    -1.960940
26  4294967297       5            rama    -0.600053
27  4294967297       5           omega     0.061867
28  4294967297       5          fa_dun     3.663050
29  4294967297       5      fa_dun_dev     0.004953
```
According to the pivot documentation, I should be able to reshape this on the score_type_name using the pivot function.
```
df.pivot(columns='score_type_name',values='score_value',index=['struct_id','resNum'])
```
But, I get the following. However, pivot_table function seems to work:
```
pivoted = df.pivot_table(columns='score_type_name',
                         values='score_value',
                         index=['struct_id','resNum'])
```
But it does not lend itself, for me atleast, to further analysis. I want it to just have the struct_id, resNum, and score_type_name as columns instead of stacking the score_type_name on top of the other columns. Additionally, I want the struct_id to be for every row, and not aggregate in a joined row like it does for the table. So can anyone tell me how I can get a nice Dataframe like I want using pivot? Additionally, from the documentation, I can't tell why pivot_table works and pivot doesn't. If I look at the first example of pivot, it looks like exactly what I need. P.S. I did post a question in reference to this problem, but I did such a poor job of demonstrating the output, I deleted it and tried again using ipython notebook. I apologize in advance if you are seeing this twice. Here is the notebook for your full reference EDIT - My desired results would look like this (made in excel):
```
StructId    resNum  pdb_residue_number  chain_id    name3   fa_dun  fa_dun_dev  fa_dun_rot  fa_dun_semi omega   p_aa_pp rama    ref
4294967297  1   99  A   ASN 2.1856  0.0000      2.1856  0.0648          -1.1912
4294967297  2   100 A   MET 1.3789  0.0286  1.3504      0.2223  -0.4425 -0.7952 0.2495
4294967297  3   101 A   VAL 0.0204  0.0255  -0.0052     0.0051  -0.0968 0.2674  0.9796
4294967297  4   102 A   GLU 4.2180  0.0037      4.2143  0.2122  -0.4628 -1.4033 -1.9609
4294967297  5   103 A   GLN 3.6630  0.0050      3.6581  0.0619  -0.2759 -0.6001 -1.5172
4294967297  6   104 A   MET 1.5175  0.2206  1.2968      0.0504  -0.3758 -0.7419 0.2495
4294967297  7   105 A   HIS 3.6987  0.0184      3.6804  0.0547  0.4019  -0.1489 0.3883
4294967297  8   106 A   THR 0.1048  0.0134  0.0914      0.0003  -0.7963 -0.4033 0.2013
4294967297  9   107 A   ASP 2.3626  0.0005      2.3620  0.0521  0.1955  -0.3499 -1.6300
4294967297  10  108 A   ILE 1.8447  0.0270  1.8176      0.0971  0.1676  -0.4071 1.0806
4294967297  11  109 A   ILE 0.1276  0.0092  0.1183      0.0208  -0.4026 -0.0075 1.0806
4294967297  12  110 A   SER 0.2921  0.0342  0.2578      0.0342  -0.2426 -1.3930 0.1654
4294967297  13  111 A   LEU 0.6483  0.0019  0.6464      0.0845  -0.3565 -0.2356 0.7611
4294967297  14  112 A   TRP 2.5965  0.1507      2.4457  0.5143  -0.1370 -0.5373 1.2341
4294967297  15  113 A   ASP 2.6448  0.1593          0.0510      -0.5011
```

Response: For anyone who is still interested in the difference between pivot and pivot_table, there are mainly two differences: pivot_table is a generalization of pivot that can handle duplicate values for one pivoted index/column pair. Specifically, you can give pivot_table a list of aggregation functions using keyword argument aggfunc. The default aggfunc of pivot_table is numpy.mean. pivot_table also supports using multiple columns for the index and column of the pivoted table. A hierarchical index will be automatically generated for you. REF: pivot and pivot_table


Question:
Pandas - add value at specific iloc into new dataframe column I have a large dataframe containing lots of columns. For each row/index in the dataframe I do some operations, read in some ancilliary ata, etc and get a new value. Is there a way to add that new value into a new column at the correct row/index? I can use .assign to add a new column but as I'm looping over the rows and only generating the data to add for one value at a time (generating it is quite involved). When it's generated I'd like to immediately add it to the dataframe rather than waiting until I've generated the entire series. This doesn't work and gives a key error:
```
df['new_column_name'].iloc[this_row]=value
```
Do I need to initialise the column first or something?

Response:
There are two steps to created & populate a new column using only a row number... (in this approach iloc is not used) First, get the row index value by using the row number
```
rowIndex = df.index[someRowNumber]
```
Then, use row index with the loc function to reference the specific row and add the new column / value
```
df.loc[rowIndex, 'New Column Title'] = "some value"
```
These two steps can be combine into one line as follows
```
df.loc[df.index[someRowNumber], 'New Column Title'] = "some value"
```


Question:
Pandas: Reading Excel with merged cells I have Excel files with multiple sheets, each of which looks a little like this (but much longer):
```
Sample  CD4     CD8
Day 1   8311    17.3    6.44
        8312    13.6    3.50
        8321    19.8    5.88
        8322    13.5    4.09
Day 2   8311    16.0    4.92
        8312    5.67    2.28
        8321    13.0    4.34
        8322    10.6    1.95
```
The first column is actually four cells merged vertically. When I read this using pandas.read_excel, I get a DataFrame that looks like this:
```
Sample    CD4   CD8
Day 1    8311  17.30  6.44
NaN      8312  13.60  3.50
NaN      8321  19.80  5.88
NaN      8322  13.50  4.09
Day 2    8311  16.00  4.92
NaN      8312   5.67  2.28
NaN      8321  13.00  4.34
NaN      8322  10.60  1.95
```
How can I either get Pandas to understand merged cells, or quickly and easily remove the NaN and group by the appropriate value? (One approach would be to reset the index, step through to find the values and replace NaNs with values, pass in the list of days, then set the index to the column. But it seems like there should be a simpler approach.)

Response:
You could use the Series.fillna method to forword-fill in the NaN values:
```
df.index = pd.Series(df.index).fillna(method='ffill')
```
For example,
```

```python
df
#Output
#Sample    CD4   CD8
#Day 1    8311  17.30  6.44
#NaN      8312  13.60  3.50
#NaN      8321  19.80  5.88
#NaN      8322  13.50  4.09
#Day 2    8311  16.00  4.92
#NaN      8312   5.67  2.28
#NaN      8321  13.00  4.34
#NaN      8322  10.60  1.95
#
#[8 rows x 3 columns]
```

```python
df.index = pd.Series(df.index).fillna(method='ffill')
```

```python
df
#Output
#Sample    CD4   CD8
#Day 1    8311  17.30  6.44
#Day 1    8312  13.60  3.50
#Day 1    8321  19.80  5.88
#Day 1    8322  13.50  4.09
#Day 2    8311  16.00  4.92
#Day 2    8312   5.67  2.28
#Day 2    8321  13.00  4.34
#Day 2    8322  10.60  1.95
#
#[8 rows x 3 columns]
#```
```


Question:
How to do a conditional count after groupby on a Pandas Dataframe? I have the following dataframe:
```none
key1  key2
0    a   one
1    a   two
2    b   one
3    b   two
4    a   one
5    c   two
```
Now, I want to group the dataframe by the key1 and count the column key2 with the value "one" to get this result:
```none
key1
0    a   2
1    b   1
2    c   0
```
I just get the usual count with:
```
df.groupby(['key1']).size()
```
But I don't know how to insert the condition. I tried things like this:
```
df.groupby(['key1']).apply(df[df['key2'] == 'one'])
```
But I can't get any further. How can I do this?

Response:
I think you need add condition first:
```
#if need also category c with no values of 'one'
df11=df.groupby('key1')['key2'].apply(lambda x: (x=='one').sum()).reset_index(name='count')
print (df11)
  key1  count
0    a      2
1    b      1
2    c      0
```
Or use categorical with key1, then missing value is added by size:
```
df['key1'] = df['key1'].astype('category')
df1 = df[df['key2'] == 'one'].groupby(['key1']).size().reset_index(name='count')
print (df1)
  key1  count
0    a      2
1    b      1
2    c      0
```
If need all combinations:
```
df2 = df.groupby(['key1', 'key2']).size().reset_index(name='count')
print (df2)
  key1 key2  count
0    a  one      2
1    a  two      1
2    b  one      1
3    b  two      1
4    c  two      1

df3 = df.groupby(['key1', 'key2']).size().unstack(fill_value=0)
print (df3)
key2  one  two
key1
a       2    1
b       1    1
c       0    1
```


Question:
How to apply "first" and "last" functions to columns while using group by in pandas? I have a data frame and I would like to group it by a particular column (or, in other words, by values from a particular column). I can do it in the following way: grouped = df.groupby(['ColumnName']). I imagine the result of this operation as a table in which some cells can contain sets of values instead of single values. To get a usual table (i.e. a table in which every cell contains only one a single value) I need to indicate what function I want to use to transform the sets of values in the cells into single values. For example I can replace sets of values by their sum, or by their minimal or maximal value. I can do it in the following way: grouped.sum() or grouped.min() and so on. Now I want to use different functions for different columns. I figured out that I can do it in the following way: grouped.agg({'ColumnName1':sum, 'ColumnName2':min}). However, because of some reasons I cannot use first. In more details, grouped.first() works, but grouped.agg({'ColumnName1':first, 'ColumnName2':first}) does not work. As a result I get a NameError: NameError: name 'first' is not defined. So, my question is: Why does it happen and how to resolve this problem. ADDED Here I found the following example:
```
grouped['D'].agg({'result1' : np.sum, 'result2' : np.mean})
```
May be I also need to use np? But in my case python does not recognize "np". Should I import it?

Response:
I think the issue is that there are two different first methods which share a name but act differently, one is for groupby objects and another for a Series/DataFrame (to do with timeseries). To replicate the behaviour of the groupby first method over a DataFrame using agg you could use iloc[0] (which gets the first row in each group (DataFrame/Series) by index):
```
grouped.agg(lambda x: x.iloc[0])
```
For example:
```

```python
df = pd.DataFrame([[1, 2], [3, 4]])
```

```python
g = df.groupby(0)
```

```python
g.first()
#Output
#1
#0
#1  2
#3  4
```

```python
g.agg(lambda x: x.iloc[0])
#Output
#1
#0
#1  2
#3  4
#```
#Analogously you can replicate last using iloc[-1]. Note: This will works column-wise, et al:
#```
#g.agg({1: lambda x: x.iloc[0]})
#```
#In older version of pandas you could would use the irow method (e.g. x.irow(0), see previous edits. A couple of updated notes: This is better done using the nth groupby method, which is much faster >=0.13:
#```
#g.nth(0)  # first
#g.nth(-1)  # last
#```
#You have to take care a little, as the default behaviour for first and last ignores NaN rows... and IIRC for DataFrame groupbys it was broken pre-0.13... there's a dropna option for nth. You can use the strings rather than built-ins (though IIRC pandas spots it's the sum builtin and applies np.sum):
#```
#grouped['D'].agg({'result1' : "sum", 'result2' : "mean"})
#```
```


Question:
Select multiple columns by labels in pandas I've been looking around for ways to select columns through the python documentation and the forums but every example on indexing columns are too simplistic. Suppose I have a 10 x 10 dataframe
```
df = DataFrame(randn(10, 10), index=range(0,10), columns=['A', 'B', 'C', 'D','E','F','G','H','I','J'])
```
So far, all the documentations gives is just a simple example of indexing like
```
subset = df.loc[:,'A':'C']
```
or
```
subset = df.loc[:,'C':]
```
But I get an error when I try index multiple, non-sequential columns, like this
```
subset = df.loc[:,('A':'C', 'E')]
```
How would I index in Pandas if I wanted to select column A to C, E, and G to I? It appears that this logic will not work
```
subset = df.loc[:,('A':'C', 'E', 'G':'I')]
```
I feel that the solution is pretty simple, but I can't get around this error. Thanks!

Response:
Name- or Label-Based (using regular expression syntax)
```
df.filter(regex='[A-CEG-I]')   # does NOT depend on the column order
```
Note that any regular expression is allowed here, so this approach can be very general. E.g. if you wanted all columns starting with a capital or lowercase "A" you could use: df.filter(regex='^[Aa]') Location-Based (depends on column order)
```
df[ list(df.loc[:,'A':'C']) + ['E'] + list(df.loc[:,'G':'I']) ]
```
Note that unlike the label-based method, this only works if your columns are alphabetically sorted. This is not necessarily a problem, however. For example, if your columns go ['A','C','B'], then you could replace 'A':'C' above with 'A':'B'. The Long Way And for completeness, you always have the option shown by @Magdalena of simply listing each column individually, although it could be much more verbose as the number of columns increases:
```
df[['A','B','C','E','G','H','I']]   # does NOT depend on the column order
```
Results for any of the above methods
```
A         B         C         E         G         H         I
0 -0.814688 -1.060864 -0.008088  2.697203 -0.763874  1.793213 -0.019520
1  0.549824  0.269340  0.405570 -0.406695 -0.536304 -1.231051  0.058018
2  0.879230 -0.666814  1.305835  0.167621 -1.100355  0.391133  0.317467
```


Question:
Create hourly/minutely time range using pandas Is there a way to generate time range in pandas similar to date_range? Something like:
```
pandas.time_range("11:00", "21:30", freq="30min")
```

Response:
A time range doesn't exist as a standalone index type. Generate using a single date
```

```python
pandas.date_range("11:00", "21:30", freq="30min")
#Output
#<class 'pandas.tseries.index.DatetimeIndex'>
#[2013-07-14 11:00:00, ..., 2013-07-14 21:30:00]
#Length: 22, Freq: 30T, Timezone: None
#```
#The time objects
#```
```

```python
pandas.date_range("11:00", "21:30", freq="30min").time
#Output
#array([datetime.time(11, 0), datetime.time(11, 30), datetime.time(12, 0),
#       datetime.time(12, 30), datetime.time(13, 0), datetime.time(13, 30),
#       datetime.time(14, 0), datetime.time(14, 30), datetime.time(15, 0),
#       datetime.time(15, 30), datetime.time(16, 0), datetime.time(16, 30),
#       datetime.time(17, 0), datetime.time(17, 30), datetime.time(18, 0),
#       datetime.time(18, 30), datetime.time(19, 0), datetime.time(19, 30),
#       datetime.time(20, 0), datetime.time(20, 30), datetime.time(21, 0),
#       datetime.time(21, 30)], dtype=object)
#```
#You can also resample if you are spanning multiple dates. What are you trying to do?
```


Question:
How do I transpose dataframe in pandas without index? Pretty sure this is very simple. I am reading a csv file and have the dataframe:
```
Attribute    A   B   C
a            1   4   7
b            2   5   8
c            3   6   9
```
I want to do a transpose to get
```
Attribute    a   b   c
A            1   2   3
B            4   5   6
C            7   8   9
```
However, when I do df.T, it results in
```
0   1   2
Attribute    a   b   c
A            1   2   3
B            4   5   6
C            7   8   9`
```
How do I get rid of the indexes on top?

Response:
You can set the index to your first column (or in general, the column you want to use as as index) in your dataframe first, then transpose the dataframe. For example if the column you want to use as index is 'Attribute', you can do:
```
df.set_index('Attribute',inplace=True)
df.transpose()
```
Or
```
df.set_index('Attribute').T
```


Question:
subsetting a Python DataFrame I am transitioning from R to Python. I just began using Pandas. I have an R code that subsets nicely:
```
k1 <- subset(data, Product = p.id & Month < mn & Year == yr, select = c(Time, Product))
```
Now, I want to do similar stuff in Python. this is what I have got so far:
```
import pandas as pd
data = pd.read_csv("../data/monthly_prod_sales.csv")

#first, index the dataset by Product. And, get all that matches a given 'p.id' and time.
 data.set_index('Product')
 k = data.ix[[p.id, 'Time']]

# then, index this subset with Time and do more subsetting..
```
I am beginning to feel that I am doing this the wrong way. perhaps, there is an elegant solution. Can anyone help? I need to extract month and year from the timestamp I have and do subsetting. Perhaps there is a one-liner that will accomplish all this:
```
k1 <- subset(data, Product = p.id & Time >= start_time & Time < end_time, select = c(Time, Product))
```
thanks.

Response:
I'll assume that Time and Product are columns in a DataFrame, df is an instance of DataFrame, and that other variables are scalar values: For now, you'll have to reference the DataFrame instance:
```
k1 = df.loc[(df.Product == p_id) & (df.Time >= start_time) & (df.Time < end_time), ['Time', 'Product']]
```
The parentheses are also necessary, because of the precedence of the & operator vs. the comparison operators. The & operator is actually an overloaded bitwise operator which has the same precedence as arithmetic operators which in turn have a higher precedence than comparison operators. In pandas 0.13 a new experimental DataFrame.query() method will be available. It's extremely similar to subset modulo the select argument: With query() you'd do it like this:
```
df[['Time', 'Product']].query('Product == p_id and Month < mn and Year == yr')
```
Here's a simple example:
```

```python
df = DataFrame({'gender': np.random.choice(['m', 'f'], size=10), 'price': poisson(100, size=10)})
```

```python
df
#Output
#gender  price
#0      m     89
#1      f    123
#2      f    100
#3      m    104
#4      m     98
#5      m    103
#6      f    100
#7      f    109
#8      f     95
#9      m     87
```

```python
df.query('gender == "m" and price < 100')
#Output
#gender  price
#0      m     89
#4      m     98
#9      m     87
#```
#The final query that you're interested will even be able to take advantage of chained comparisons, like this:
#```
#k1 = df[['Time', 'Product']].query('Product == p_id and start_time <= Time < end_time')
#```
```


Question:
'utf-8' codec can't decode byte 0x92 in position 18: invalid start byte I am trying to read in a dataset called df1, but it does not work
```
import pandas as pd
df1=pd.read_csv("https://raw.githubusercontent.com/tuyenhavan/Statistics/Dataset/World_Life_Expectancy.csv",sep=";")

df1.head()
```
Here are huge errors from the above code, but this is the most relevant
```
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 18: invalid start byte
```

Response:
The data is indeed not encoded as UTF-8; everything is ASCII except for that single 0x92 byte:
```
b'Korea, Dem. People\x92s Rep.'
```
Decode it as Windows codepage 1252 instead, where 0x92 is a fancy quote, ’:
```
df1 = pd.read_csv("https://raw.githubusercontent.com/tuyenhavan/Statistics/Dataset/World_Life_Expectancy.csv",
                  sep=";", encoding='cp1252')
```
Demo:
```

```python
import pandas as pd
```

```python
df1 = pd.read_csv("https://raw.githubusercontent.com/tuyenhavan/Statistics/Dataset/World_Life_Expectancy.csv",
                  sep=";", encoding='cp1252')
```

```python
df1.head()
#Output
#                   2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  \
#0     Afghanistan  55.1  55.5  55.9  56.2  56.6  57.0  57.4  57.8  58.2  58.6
#1         Albania  74.3  74.7  75.2  75.5  75.8  76.1  76.3  76.5  76.7  76.8
#2         Algeria  70.2  70.6  71.0  71.4  71.8  72.2  72.6  72.9  73.2  73.5
#3  American Samoa    ..    ..    ..    ..    ..    ..    ..    ..    ..    ..
#4         Andorra    ..    ..    ..    ..    ..    ..    ..    ..    ..    ..
```

2010  2011  2012  2013  Unnamed: 15  2014  2015
0  59.0  59.3  59.7  60.0          NaN  60.4  60.7
1  77.0  77.2  77.4  77.6          NaN  77.8  78.0
2  73.8  74.1  74.3  74.6          NaN  74.8  75.0
3    ..    ..    ..    ..          NaN    ..    ..
4    ..    ..    ..    ..          NaN    ..    ..
```
I note however, that Pandas seems to take the HTTP headers at face value too and produces a Mojibake when you load your data from a URL. When I save the data directly to disk, then load it with pd.read_csv() the data is correctly decoded, but loading from the URL produces re-coded data:
```

```python
df1[' '][102]
#Output
#'Korea, Dem. Peopleâ€™s Rep.'
```

```python
df1[' '][102].encode('cp1252').decode('utf8')
#Output
#'Korea, Dem. People’s Rep.'
#```
#This is a known bug in Pandas. You can work around this by using urllib.request to load the URL and pass that to pd.read_csv() instead:
#```
```

```python
import urllib.request
```

```python
with urllib.request.urlopen("https://raw.githubusercontent.com/tuyenhavan/Statistics/Dataset/World_Life_Expectancy.csv") as resp:
    df1 = pd.read_csv(resp, sep=";", encoding='cp1252')
#Output
#...
```

```python
df1[' '][102]
#Output
#'Korea, Dem. People’s Rep.'
#```
```


Question:
Selecting Pandas Columns by dtype I was wondering if there is an elegant and shorthand way in Pandas DataFrames to select columns by data type (dtype). i.e. Select only int64 columns from a DataFrame. To elaborate, something along the lines of
```
df.select_columns(dtype=float64)
```

Response:
Since 0.14.1 there's a select_dtypes method so you can do this more elegantly/generally.
```

```python
df = pd.DataFrame([[1, 2.2, 'three']], columns=['A', 'B', 'C'])
```

```python
df.select_dtypes(include=['int'])
#Output
#A
#0  1
#```
#To select all numeric types use the numpy dtype numpy.number
#```
```

```python
df.select_dtypes(include=[np.number])
#Output
#A    B
#0  1  2.2
```

```python
df.select_dtypes(exclude=[object])
#Output
#A    B
#0  1  2.2
#```
```


Question:
Convert whole dataframe from lower case to upper case with Pandas I have a dataframe like the one displayed below:
```
# Create an example dataframe about a fictional army
raw_data = {'regiment': ['Nighthawks', 'Nighthawks', 'Nighthawks', 'Nighthawks'],
            'company': ['1st', '1st', '2nd', '2nd'],
            'deaths': ['kkk', 52, '25', 616],
            'battles': [5, '42', 2, 2],
            'size': ['l', 'll', 'l', 'm']}
df = pd.DataFrame(raw_data, columns = ['regiment', 'company', 'deaths', 'battles', 'size'])
```
My goal is to transform every single string inside of the dataframe to upper case so that it looks like this: Notice: all data types are objects and must not be changed; the output must contain all objects. I want to avoid to convert every single column one by one... I would like to do it generally over the whole dataframe possibly. What I tried so far is to do this but without success
```
df.str.upper()
```

Response:
astype() will cast each series to the dtype object (string) and then call the str() method on the converted series to get the string literally and call the function upper() on it. Note that after this, the dtype of all columns changes to object.
```

```python
df
#Output
#regiment company deaths battles size
#0  Nighthawks     1st    kkk       5    l
#1  Nighthawks     1st     52      42   ll
#2  Nighthawks     2nd     25       2    l
#3  Nighthawks     2nd    616       2    m
```

```python
df.apply(lambda x: x.astype(str).str.upper())
#Output
#regiment company deaths battles size
#0  NIGHTHAWKS     1ST    KKK       5    L
#1  NIGHTHAWKS     1ST     52      42   LL
#2  NIGHTHAWKS     2ND     25       2    L
#3  NIGHTHAWKS     2ND    616       2    M
#```
#You can later convert the 'battles' column to numeric again, using to_numeric():
#```
```

```python
df2 = df.apply(lambda x: x.astype(str).str.upper())
```

```python
df2['battles'] = pd.to_numeric(df2['battles'])
```

```python
df2
#Output
#regiment company deaths  battles size
#0  NIGHTHAWKS     1ST    KKK        5    L
#1  NIGHTHAWKS     1ST     52       42   LL
#2  NIGHTHAWKS     2ND     25        2    L
#3  NIGHTHAWKS     2ND    616        2    M
```

```python
df2.dtypes
#Output
#regiment    object
#company     object
#deaths      object
#battles      int64
#size        object
#dtype: object
#```
```


Question:
Rename result columns from Pandas aggregation ("FutureWarning: using a dict with renaming is deprecated") I'm trying to do some aggregations on a pandas data frame. Here is a sample code:
```
import pandas as pd

df = pd.DataFrame({"User": ["user1", "user2", "user2", "user3", "user2", "user1"],
                  "Amount": [10.0, 5.0, 8.0, 10.5, 7.5, 8.0]})

df.groupby(["User"]).agg({"Amount": {"Sum": "sum", "Count": "count"}})

Out[1]:
      Amount
         Sum Count
User
user1   18.0     2
user2   20.5     3
user3   10.5     1
```
Which generates the following warning: FutureWarning: using a dict with renaming is deprecated and will be removed in a future version return super(DataFrameGroupBy, self).aggregate(arg, *args, **kwargs) How can I avoid this?

Response:
Use groupby apply and return a Series to rename columns Use the groupby apply method to perform an aggregation that Renames the columns Allows for spaces in the names Allows you to order the returned columns in any way you choose Allows for interactions between columns Returns a single level index and NOT a MultiIndex To do this: create a custom function that you pass to apply This custom function is passed each group as a DataFrame Return a Series The index of the Series will be the new columns Create fake data
```
df = pd.DataFrame({"User": ["user1", "user2", "user2", "user3", "user2", "user1", "user3"],
                  "Amount": [10.0, 5.0, 8.0, 10.5, 7.5, 8.0, 9],
                  'Score': [9, 1, 8, 7, 7, 6, 9]})
```
create custom function that returns a Series The variable x inside of my_agg is a DataFrame
```
def my_agg(x):
    names = {
        'Amount mean': x['Amount'].mean(),
        'Amount std':  x['Amount'].std(),
        'Amount range': x['Amount'].max() - x['Amount'].min(),
        'Score Max':  x['Score'].max(),
        'Score Sum': x['Score'].sum(),
        'Amount Score Sum': (x['Amount'] * x['Score']).sum()}

    return pd.Series(names, index=['Amount range', 'Amount std', 'Amount mean',
                                   'Score Sum', 'Score Max', 'Amount Score Sum'])
```
Pass this custom function to the groupby apply method
```
df.groupby('User').apply(my_agg)
```
The big downside is that this function will be much slower than agg for the cythonized aggregations Using a dictionary with groupby agg method Using a dictionary of dictionaries was removed because of its complexity and somewhat ambiguous nature. There is an ongoing discussion on how to improve this functionality in the future on github Here, you can directly access the aggregating column after the groupby call. Simply pass a list of all the aggregating functions you wish to apply.
```
df.groupby('User')['Amount'].agg(['sum', 'count'])
```
Output
```
sum  count
User
user1  18.0      2
user2  20.5      3
user3  10.5      1
```
It is still possible to use a dictionary to explicitly denote different aggregations for different columns, like here if there was another numeric column named Other.
```
df = pd.DataFrame({"User": ["user1", "user2", "user2", "user3", "user2", "user1"],
              "Amount": [10.0, 5.0, 8.0, 10.5, 7.5, 8.0],
              'Other': [1,2,3,4,5,6]})

df.groupby('User').agg({'Amount' : ['sum', 'count'], 'Other':['max', 'std']})
```
Output
```
Amount       Other
         sum count   max       std
User
user1   18.0     2     6  3.535534
user2   20.5     3     5  1.527525
user3   10.5     1     4       NaN
```


Question:
Max and Min date in pandas groupby I have a dataframe that looks like:
```
data = {'index': ['2014-06-22 10:46:00', '2014-06-24 19:52:00', '2014-06-25 17:02:00', '2014-06-25 17:55:00', '2014-07-02 11:36:00', '2014-07-06 12:40:00', '2014-07-05 12:46:00', '2014-07-27 15:12:00'],
    'type': ['A', 'B', 'C', 'A', 'B', 'C', 'A', 'C'],
    'sum_col': [1, 2, 3, 1, 1, 3, 2, 1]}
df = pd.DataFrame(data, columns=['index', 'type', 'sum_col'])
df['index'] = pd.to_datetime(df['index'])
df = df.set_index('index')
df['weekofyear'] = df.index.weekofyear
df['date'] = df.index.date
df['date'] = pd.to_datetime(df['date'])

                     type sum_col weekofyear   date
index
2014-06-22 10:46:00    A    1       25      2014-06-22
2014-06-24 19:52:00    B    2       26      2014-06-24
2014-06-25 17:02:00    C    3       26      2014-06-25
2014-06-25 17:55:00    A    1       26      2014-06-25
2014-07-02 11:36:00    B    1       27      2014-07-02
2014-07-06 12:40:00    C    3       27      2014-07-06
2014-07-05 12:46:00    A    2       27      2014-07-05
2014-07-27 15:12:00    C    1       30      2014-07-27
```
I'm looking to groupby the weekofyear, then sum up the sum_col. In addition, I need to find the earliest, and the latest date for the week. The first part is pretty easy:
```
gb = df.groupby(['type', 'weekofyear'])
gb['sum_col'].agg({'sum_col' : np.sum})
```
I've tried to find the min/max date with this, but haven't been successful:
```
gb = df.groupby(['type', 'weekofyear'])
gb.agg({'sum_col' : np.sum,
        'date' : np.min,
        'date' : np.max})
```
How would one find the earliest/latest date that appears?

Response:
You need to combine the functions that apply to the same column, like this:
```

```python
gb.agg({'sum_col' : np.sum,
     ...:         'date' : [np.min, np.max]})
#Output
#date             sum_col
#                      amin       amax      sum
#type weekofyear
#A    25         2014-06-22 2014-06-22        1
#     26         2014-06-25 2014-06-25        1
#     27         2014-07-05 2014-07-05        2
#B    26         2014-06-24 2014-06-24        2
#     27         2014-07-02 2014-07-02        1
#C    26         2014-06-25 2014-06-25        3
#     27         2014-07-06 2014-07-06        3
#     30         2014-07-27 2014-07-27        1
#```
```


Question:
pandas groupby and join lists I have a dataframe df, with two columns, I want to groupby one column and join the lists belongs to same group, example:
```
column_a, column_b
1,         [1,2,3]
1,         [2,5]
2,         [5,6]
```
after the process:
```
column_a, column_b
1,         [1,2,3,2,5]
2,         [5,6]
```
I want to keep all the duplicates. I have the following questions: The dtypes of the dataframe are object(s). convert_objects() doesn't convert column_b to list automatically. How can I do this? what does the function in df.groupby(...).apply(lambda x: ...) apply to ? what is the form of x ? list? the solution to my main problem? Thanks in advance.

Response:
object dtype is a catch-all dtype that basically means not int, float, bool, datetime, or timedelta. So it is storing them as a list. convert_objects tries to convert a column to one of those dtypes. You want
```

```python
df
#Output
#a          b    c
#0  1  [1, 2, 3]  foo
#1  1     [2, 5]  bar
#2  2     [5, 6]  baz
```

```python
df.groupby('a').agg({'b': 'sum', 'c': lambda x: ' '.join(x)})
#Output
#c                b
#a
#1  foo bar  [1, 2, 3, 2, 5]
#2      baz           [5, 6]
#```
#This groups the data frame by the values in column a. Read more about groupby. This is doing a regular list sum (concatenation) just like [1, 2, 3] + [2, 5] with the result [1, 2, 3, 2, 5]
```


Question:
How to check if a variable is either a python list, numpy array or pandas series I have a function that takes in a variable that would work if it is any of the following three types
```
1. pandas Series
 2. numpy array (ndarray)
 3. python list
```
Any other type should be rejected. What is the most efficient way to check this?

Response:
You can do it using isinstance:
```
import pandas as pd
import numpy as np
def f(l):
    if isinstance(l,(list,pd.core.series.Series,np.ndarray)):
        print(5)
    else:
        raise Exception('wrong type')
```
Then f([1,2,3]) prints 5 while f(3.34) raises an error.


Question:
better way to drop nan rows in pandas On my own I found a way to drop nan rows from a pandas dataframe. Given a dataframe dat with column x which contains nan values,is there a more elegant way to do drop each row of dat which has a nan value in the x column?
```
dat = dat[np.logical_not(np.isnan(dat.x))]
dat = dat.reset_index(drop=True)
```

Response:
Use dropna:
```
dat.dropna()
```
You can pass param how to drop if all labels are nan or any of the labels are nan
```
dat.dropna(how='any')    #to drop if any value in the row has a nan
dat.dropna(how='all')    #to drop if all values in the row are nan
```
Hope that answers your question! Edit 1: In case you want to drop rows containing nan values only from particular column(s), as suggested by J. Doe in his answer below, you can use the following:
```
dat.dropna(subset=[col_list])  # col_list is a list of column names to consider for nan values.
```


Question:
Python pandas: how to remove nan and -inf values I have the following dataframe
```
time       X    Y  X_t0     X_tp0  X_t1     X_tp1  X_t2     X_tp2
0         0.002876    0   10     0       NaN   NaN       NaN   NaN       NaN
1         0.002986    0   10     0       NaN     0       NaN   NaN       NaN
2         0.037367    1   10     1  1.000000     0       NaN     0       NaN
3         0.037374    2   10     2  0.500000     1  1.000000     0       NaN
4         0.037389    3   10     3  0.333333     2  0.500000     1  1.000000
5         0.037393    4   10     4  0.250000     3  0.333333     2  0.500000

....
1030308   9.962213  256  268   256  0.000000   256  0.003906   255  0.003922
1030309  10.041799    0  268     0      -inf   256  0.000000   256  0.003906
1030310  10.118960    0  268     0       NaN     0      -inf   256  0.000000
```
I tried with the following
```
df.dropna(inplace=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.40)

X_train = X_train.drop('time', axis=1)
X_train = X_train.drop('X_t1', axis=1)
X_train = X_train.drop('X_t2', axis=1)
X_test = X_test.drop('time', axis=1)
X_test = X_test.drop('X_t1', axis=1)
X_test = X_test.drop('X_t2', axis=1)
X_test.fillna(X_test.mean(), inplace=True)
X_train.fillna(X_train.mean(), inplace=True)
y_train.fillna(y_train.mean(), inplace=True)
```
However, I am still getting this error ValueError: Input contains NaN, infinity or a value too large for dtype('float32'). whenever i try to fit a regression model fit(X_train, y_train) How can we remove both the NaN and -inf values at the same time?

Response:
Use pd.DataFrame.isin and check for rows that have any with pd.DataFrame.any. Finally, use the boolean array to slice the dataframe.
```
df[~df.isin([np.nan, np.inf, -np.inf]).any(1)]

             time    X    Y  X_t0     X_tp0   X_t1     X_tp1   X_t2     X_tp2
4        0.037389    3   10     3  0.333333    2.0  0.500000    1.0  1.000000
5        0.037393    4   10     4  0.250000    3.0  0.333333    2.0  0.500000
1030308  9.962213  256  268   256  0.000000  256.0  0.003906  255.0  0.003922
```


Question:
assigning column names to a pandas series I have a pandas series
```
object x
Ezh2   2
Hmgb   7
Irf1   1
```
I want to save this as a dataframe with column names Gene and Count respectively I tried
```
x_df = pd.DataFrame(x,columns = ['Gene','count'])
```
but it does not work.The final form I want is
```
Gene Count
Ezh2   2
Hmgb   7
Irf1   1
```
Can you suggest how to do this

Response:
You can create a dict and pass this as the data param to the dataframe constructor:
```

```python
df = pd.DataFrame({'Gene':s.index, 'count':s.values})
df
#Output
#Gene  count
#0  Ezh2      2
#1  Hmgb      7
#2  Irf1      1
#```
#Alternatively you can create a df from the series, you need to call reset_index as the index will be used and then rename the columns:
#```
```

```python
df = pd.DataFrame(s).reset_index()
df.columns = ['Gene', 'count']
df
#Output
#Gene  count
#0  Ezh2      2
#1  Hmgb      7
#2  Irf1      1
#```
```


Question:
How can I partially read a huge CSV file? I have a very big csv file so that I can not read it all into memory. I only want to read and process a few lines in it. So I am seeking a function in Pandas which could handle this task, which basic python can handle well:
```
with open('abc.csv') as f:
    line = f.readline()
    # pass until it reaches a particular line number....
```
However, if I do this in pandas, I always read the first line:
```
datainput1 = pd.read_csv('matrix.txt', sep=',', header=None, nrows=1)
datainput2 = pd.read_csv('matrix.txt', sep=',', header=None, nrows=1)
```
I am looking for some easier way to handle this task in pandas. For example, if I want to read rows from 1000 to 2000. How can I do this quickly? I want to use pandas because I want to read data into the dataframe.

Response:
Use chunksize:
```
for df in pd.read_csv('matrix.txt', sep=',', header=None, chunksize=1):
    #do something
```
To answer your second part do this:
```
df = pd.read_csv('matrix.txt', sep=',', header=None, skiprows=1000, chunksize=1000)
```
This will skip the first 1000 rows and then only read the next 1000 rows giving you rows 1000-2000, unclear if you require the end points to be included or not but you can fiddle the numbers to get what you want.


Question:
Replace value for a selected cell in pandas DataFrame without using index this is a rather similar question to this question but with one key difference: I'm selecting the data I want to change not by its index but by some criteria. If the criteria I apply return a single row, I'd expect to be able to set the value of a certain column in that row in an easy way, but my first attempt doesn't work:
```

```python
d = pd.DataFrame({'year':[2008,2008,2008,2008,2009,2009,2009,2009],
                  'flavour':['strawberry','strawberry','banana','banana',
                  'strawberry','strawberry','banana','banana'],
                  'day':['sat','sun','sat','sun','sat','sun','sat','sun'],
                  'sales':[10,12,22,23,11,13,23,24]})
```

```python
d
#Output
#   day     flavour  sales  year
#0  sat  strawberry     10  2008
#1  sun  strawberry     12  2008
#2  sat      banana     22  2008
#3  sun      banana     23  2008
#4  sat  strawberry     11  2009
#5  sun  strawberry     13  2009
#6  sat      banana     23  2009
#7  sun      banana     24  2009
```

```python
d[d.sales==24]
#Output
#   day flavour  sales  year
#7  sun  banana     24  2009
```

```python
d[d.sales==24].sales = 100
```

```python
d
#Output
#   day     flavour  sales  year
#0  sat  strawberry     10  2008
#1  sun  strawberry     12  2008
#2  sat      banana     22  2008
#3  sun      banana     23  2008
#4  sat  strawberry     11  2009
#5  sun  strawberry     13  2009
#6  sat      banana     23  2009
#7  sun      banana     24  2009
#```
#So rather than setting 2009 Sunday's Banana sales to 100, nothing happens! What's the nicest way to do this? Ideally the solution should use the row number, as you normally don't know that in advance!
```

Response:
Many ways to do that 1
```

```python
d.sales[d.sales==24] = 100
```

```python
d
#Output
#day     flavour  sales  year
#0  sat  strawberry     10  2008
#1  sun  strawberry     12  2008
#2  sat      banana     22  2008
#3  sun      banana     23  2008
#4  sat  strawberry     11  2009
#5  sun  strawberry     13  2009
#6  sat      banana     23  2009
#7  sun      banana    100  2009
#```
#2
#```
```

```python
d.loc[d.sales == 12, 'sales'] = 99
```

```python
d
#Output
#day     flavour  sales  year
#0  sat  strawberry     10  2008
#1  sun  strawberry     99  2008
#2  sat      banana     22  2008
#3  sun      banana     23  2008
#4  sat  strawberry     11  2009
#5  sun  strawberry     13  2009
#6  sat      banana     23  2009
#7  sun      banana    100  2009
#```
#3
#```
```

```python
d.sales = d.sales.replace(23, 24)
```

```python
d
#Output
#day     flavour  sales  year
#0  sat  strawberry     10  2008
#1  sun  strawberry     99  2008
#2  sat      banana     22  2008
#3  sun      banana     24  2008
#4  sat  strawberry     11  2009
#5  sun  strawberry     13  2009
#6  sat      banana     24  2009
#7  sun      banana    100  2009
#```
```


Question:
Finding count of distinct elements in DataFrame in each column I am trying to find the count of distinct values in each column using Pandas. This is what I did.
```
import pandas as pd
import numpy as np

# Generate data.
NROW = 10000
NCOL = 100
df = pd.DataFrame(np.random.randint(1, 100000, (NROW, NCOL)),
                  columns=['col' + x for x in np.arange(NCOL).astype(str)])
```
I need to count the number of distinct elements for each column, like this:
```
col0    9538
col1    9505
col2    9524
```
What would be the most efficient way to do this, as this method will be applied to files which have size greater than 1.5GB? Based upon the answers, df.apply(lambda x: len(x.unique())) is the fastest (notebook). %timeit df.apply(lambda x: len(x.unique())) 10 loops, best of 3: 49.5 ms per loop %timeit df.nunique() 10 loops, best of 3: 59.7 ms per loop %timeit df.apply(pd.Series.nunique) 10 loops, best of 3: 60.3 ms per loop %timeit df.T.apply(lambda x: x.nunique(), axis=1) 10 loops, best of 3: 60.5 ms per loop

Response:
As of pandas 0.20 we can use nunique directly on DataFrames, i.e.:
```
df.nunique()
a    4
b    5
c    1
dtype: int64
```
Other legacy options: You could do a transpose of the df and then using apply call nunique row-wise:
```

```python
df = pd.DataFrame({'a':[0,1,1,2,3],'b':[1,2,3,4,5],'c':[1,1,1,1,1]})
df
#Output
#a  b  c
#0  0  1  1
#1  1  2  1
#2  1  3  1
#3  2  4  1
#4  3  5  1
```

```python
df.T.apply(lambda x: x.nunique(), axis=1)
#Output
#a    4
#b    5
#c    1
#dtype: int64
#```
#EDIT As pointed out by @ajcr the transpose is unnecessary:
#```
```

```python
df.apply(pd.Series.nunique)
#Output
#a    4
#b    5
#c    1
#dtype: int64
#```
```


Question:
Pandas Dataframe: split column into multiple columns, right-align inconsistent cell entries I have a pandas dataframe with a column named 'City, State, Country'. I want to separate this column into three new columns, 'City, 'State' and 'Country'.
```
0                 HUN
1                 ESP
2                 GBR
3                 ESP
4                 FRA
5             ID, USA
6             GA, USA
7    Hoboken, NJ, USA
8             NJ, USA
9                 AUS
```
Splitting the column into three columns is trivial enough:
```
location_df = df['City, State, Country'].apply(lambda x: pd.Series(x.split(',')))
```
However, this creates left-aligned data:
```
0       1       2
0    HUN     NaN     NaN
1    ESP     NaN     NaN
2    GBR     NaN     NaN
3    ESP     NaN     NaN
4    FRA     NaN     NaN
5    ID      USA     NaN
6    GA      USA     NaN
7    Hoboken  NJ     USA
8    NJ      USA     NaN
9    AUS     NaN     NaN
```
How would one go about creating the new columns with the data right-aligned? Would I need to iterate through every row, count the number of commas and handle the contents individually?

Response:
I'd do something like the following:
```
foo = lambda x: pd.Series([i for i in reversed(x.split(','))])
rev = df['City, State, Country'].apply(foo)
print rev

      0    1        2
0   HUN  NaN      NaN
1   ESP  NaN      NaN
2   GBR  NaN      NaN
3   ESP  NaN      NaN
4   FRA  NaN      NaN
5   USA   ID      NaN
6   USA   GA      NaN
7   USA   NJ  Hoboken
8   USA   NJ      NaN
9   AUS  NaN      NaN
```
I think that gets you what you want but if you also want to pretty things up and get a City, State, Country column order, you could add the following:
```
rev.rename(columns={0:'Country',1:'State',2:'City'},inplace=True)
rev = rev[['City','State','Country']]
print rev

     City State Country
0      NaN   NaN     HUN
1      NaN   NaN     ESP
2      NaN   NaN     GBR
3      NaN   NaN     ESP
4      NaN   NaN     FRA
5      NaN    ID     USA
6      NaN    GA     USA
7  Hoboken    NJ     USA
8      NaN    NJ     USA
9      NaN   NaN     AUS
```