Question:
How to delete the last column of data of a pandas dataframe I have some cvs data that has an empty column at the end of each row. I would like to leave it out of the import or alternatively delete it after import. My cvs data's have a varying number of columns. I've tried using df.tail(), but haven't managed to choose the last column with it.
```
employment=pd.read_csv('./data/spanish/employment1976-1987thousands.csv',index_col=0,header=[7,8],encoding='latin-1')
```
Data:
```
4.- Resultados provinciales
Encuesta de Población Activa. Principales Resultados

Activos por provincia y grupo de edad (4).
Unidades:miles de personas

,Álava,,,,Albacete,,,,Alicante,,,,Almería,,,,Asturias,,,,Ávila,,,,Badajoz,,,,Balears (Illes),,,,Barcelona,,,,Burgos,,,,Cáceres,,,,Cádiz,,,,Cantabria,,,,Castellón de la Plana,,,,Ciudad Real,,,,Córdoba,,,,Coruña (A),,,,Cuenca,,,,Girona,,,,Granada,,,,Guadalajara,,,,Guipúzcoa,,,,Huelva,,,,Huesca,,,,Jaén,,,,León,,,,Lleida,,,,Lugo,,,,Madrid,,,,Málaga,,,,Murcia,,,,Navarra,,,,Orense,,,,Palencia,,,,Palmas (Las),,,,Pontevedra,,,,Rioja (La),,,,Salamanca,,,,Santa Cruz de Tenerife,,,,Segovia,,,,Sevilla,,,,Soria,,,,Tarragona,,,,Teruel,,,,Toledo,,,,Valencia,,,,Valladolid,,,,Vizcaya,,,,Zamora,,,,Zaragoza,,,,Ceuta y Melilla,,,,
,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,de 16 a 19 años,de 20 a 24 años,de 25 a 54 años,de 55 y más años,
1976TIII,"8.9","11.6","60.4","11.8","16.4","14.4","65.2","14.9","47.9","49.9","246.0","60.1","20.5","14.3","88.9","11.2","34.5","42.5","278.0","91.3","6.6","7.2","41.5","13.3","25.3","22.8","135.3","37.5","19.8","24.4","153.0","43.0","166.8","203.7","1079.0","230.7","14.1","16.4","86.0","23.8","17.0","18.3","86.6","28.6","31.0","38.7","180.4","29.8","15.3","19.2","120.6","30.4","19.9","15.3","104.2","23.4","19.7","19.5","97.5","29.7","28.0","23.9","140.5","30.1","29.1","46.1","263.8","70.0","8.9","6.2","45.7","14.6","19.7","19.7","123.0","35.3","26.8","22.5","141.0","36.2","4.8","6.0","33.1","13.4","23.1","31.6","174.5","33.8","11.9","14.3","83.8","18.8","7.0","9.3","50.3","20.0","22.4","23.4","125.8","28.6","22.7","21.6","143.1","50.9","12.5","13.7","89.5","33.2","14.3","14.7","134.0","54.7","136.6","207.5","1067.6","218.6","34.7","41.1","196.4","38.4","37.2","35.0","200.5","46.1","15.6","23.8","111.6","30.7","14.0","16.8","120.2","74.9","5.7","6.4","39.2","8.0","24.5","25.6","135.3","27.1","36.4","39.4","246.1","74.0","10.2","11.3","63.9","13.4","10.5","11.0","74.1","19.6","19.3","23.9","140.3","31.7","5.5","6.0","35.6","11.3","55.2","55.6","262.5","68.1","3.1","3.2","24.4","5.4","21.8","18.4","116.7","37.1","4.6","3.4","37.3","12.0","20.3","16.7","102.2","23.1","73.5","85.5","454.6","101.5","19.2","23.4","90.7","20.5","41.3","54.7","272.2","57.0","6.0","7.1","56.5","28.9","29.2","32.1","192.7","49.8","0.0","0.0","0.0","0.0",
1976TIV,"8.7","11.7","60.8","11.4","14.4","13.6","63.3","14.5","49.1","50.6","244.9","54.2","19.0","16.9","86.8","11.4","33.2","42.3","271.8","86.0","5.8","7.5","40.3","13.9","25.1","24.7","132.7","38.4","18.8","23.4","151.8","43.9","172.2","201.7","1070.7","228.1","11.1","15.7","82.5","21.1","16.4","18.0","89.2","26.6","32.6","40.0","176.5","30.5","15.8","18.1","121.3","30.2","19.0","17.3","106.3","24.1","19.9","19.0","101.7","26.9","25.3","22.3","142.7","28.9","30.0","42.4","267.6","70.1","7.3","7.0","44.4","13.0","17.8","21.4","122.8","34.0","28.4","21.6","140.5","36.8","4.7","6.6","32.6","10.8","24.8","32.7","177.2","32.3","11.9","12.5","85.4","20.5","6.9","8.5","48.8","19.9","22.4","22.1","127.6","25.1","18.5","21.1","137.8","48.7","12.4","11.1","84.9","31.5","13.6","15.6","132.7","52.0","144.0","202.3","1054.0","222.5","35.6","40.1","194.1","37.5","36.7","34.7","203.8","47.1","15.6","23.6","114.3","31.3","14.0","15.9","118.3","76.7","5.5","7.3","36.9","9.3","25.5","25.1","138.7","26.8","34.8","42.9","250.3","74.9","9.9","11.8","62.8","14.0","10.0","13.2","74.5","19.2","19.5","24.2","142.7","31.0","4.0","5.9","35.5","12.0","55.0","56.7","264.7","63.3","2.8","3.5","23.9","5.1","20.0","21.6","116.4","34.9","4.5","3.7","36.5","12.1","21.1","17.6","100.6","25.7","74.6","87.5","455.5","102.1","18.9","22.9","90.0","21.6","40.2","57.1","273.9","58.5","5.6","8.3","57.6","23.9","28.3","31.4","192.2","46.4","0.0","0.0","0.0","0.0",
1977TI,"9.2","11.8","59.9","11.2","14.2","13.2","65.9","14.7","48.2","50.4","251.1","50.8","17.8","15.4","86.5","11.8","30.6","42.9","272.6","84.1","5.8","7.4","37.2","12.8","24.1","22.8","131.3","38.2","17.8","23.5","151.1","42.5","168.1","200.4","1077.2","223.3","11.6","12.8","80.9","17.6","14.4","16.4","88.2","23.9","34.5","37.5","176.3","30.8","15.2","19.7","121.3","31.6","18.4","19.4","107.4","24.7","20.0","18.1","98.3","26.6","24.9","23.6","150.7","27.5","29.5","40.3","267.4","70.5","5.6","7.5","44.2","12.8","17.1","21.1","122.8","33.6","29.6","23.3","142.1","37.9","4.6","5.5","33.7","11.2","23.5","30.4","175.2","32.8","12.0","12.7","84.8","21.3","7.3","9.3","46.6","17.8","30.2","26.0","147.1","25.2","15.9","22.7","133.2","45.1","12.8","12.1","84.3","28.0","12.4","16.5","131.2","55.6","150.9","202.9","1065.4","223.7","36.6","44.0","194.3","39.9","36.7","31.5","196.7","45.7","14.8","22.5","115.1","29.4","11.7","17.2","114.2","75.8","5.0","7.7","38.0","9.4","24.0","26.8","143.5","27.0","35.3","43.0","247.4","73.5","9.7","12.1","61.6","13.3","9.5","11.9","73.9","18.9","20.4","26.7","143.0","31.6","4.0","5.0","35.5","12.3","52.3","58.0","266.0","62.5","2.6","2.7","24.2","6.0","17.3","21.0","113.0","33.3","4.5","5.2","33.8","10.6","18.7","18.8","98.3","24.8","77.4","87.6","446.6","100.3","20.5","23.4","90.2","20.4","38.7","50.7","277.6","57.3","6.4","8.7","60.1","21.5","28.6","31.0","194.8","45.7","0.0","0.0","0.0","0.0",
```

Response:
Here's a one-liner that does not require specifying the column name
```
df.drop(df.columns[len(df.columns)-1], axis=1, inplace=True)
```


Question:
Conditional Logic on Pandas DataFrame [duplicate] This question already has answers here: Create new column based on values from other columns / apply a function of multiple columns, row-wise in Pandas (8 answers) How do I create a new column where the values are selected based on an existing column? (15 answers) Closed last year. How to apply conditional logic to a Pandas DataFrame. See DataFrame shown below,
```
data desired_output
0     1          False
1     2          False
2     3           True
3     4           True
```
My original data is show in the 'data' column and the desired_output is shown next to it. If the number in 'data' is below 2.5, the desired_output is False. I could apply a loop and do re-construct the DataFrame... but that would be 'un-pythonic'

Response:
```

```python
df
#Output
#data
#0     1
#1     2
#2     3
#3     4
#```
#You want to apply a function that conditionally returns a value based on the selected dataframe column.
#```
```

```python
df['data'].apply(lambda x: 'true' if x <= 2.5 else 'false')
#Output
#0     true
#1     true
#2    false
#3    false
#Name: data
#```
#You can then assign that returned column to a new column in your dataframe:
#```
```

```python
df['desired_output'] = df['data'].apply(lambda x: 'true' if x <= 2.5 else 'false')
```

```python
df
#Output
#data desired_output
#0     1           true
#1     2           true
#2     3          false
#3     4          false
#```
```


Question:
Converting statsmodels summary object to Pandas Dataframe I am doing multiple linear regression with statsmodels.formula.api (ver 0.9.0) on Windows 10. After fitting the model and getting the summary with following lines i get summary in summary object format.
```
X_opt  = X[:, [0,1,2,3]]
regressor_OLS = sm.OLS(endog= y, exog= X_opt).fit()
regressor_OLS.summary()

                          OLS Regression Results
==============================================================================
Dep. Variable:                      y   R-squared:                       0.951
Model:                            OLS   Adj. R-squared:                  0.948
Method:                 Least Squares   F-statistic:                     296.0
Date:                Wed, 08 Aug 2018   Prob (F-statistic):           4.53e-30
Time:                        00:46:48   Log-Likelihood:                -525.39
No. Observations:                  50   AIC:                             1059.
Df Residuals:                      46   BIC:                             1066.
Df Model:                           3
Covariance Type:            nonrobust
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const       5.012e+04   6572.353      7.626      0.000    3.69e+04    6.34e+04
x1             0.8057      0.045     17.846      0.000       0.715       0.897
x2            -0.0268      0.051     -0.526      0.602      -0.130       0.076
x3             0.0272      0.016      1.655      0.105      -0.006       0.060
==============================================================================
Omnibus:                       14.838   Durbin-Watson:                   1.282
Prob(Omnibus):                  0.001   Jarque-Bera (JB):               21.442
Skew:                          -0.949   Prob(JB):                     2.21e-05
Kurtosis:                       5.586   Cond. No.                     1.40e+06
==============================================================================
```
I want to do backward elimination for P values for significance level 0.05. For this i need to remove the predictor with highest P values and run the code again. I wanted to know if there is a way to extract the P values from the summary object, so that i can run a loop with conditional statement and find the significant variables without repeating the steps manually. Thank you.

Response:
The answer from @Michael B works well, but requires "recreating" the table. The table itself is actually directly available from the summary().tables attribute. Each table in this attribute (which is a list of tables) is a SimpleTable, which has methods for outputting different formats. We can then read any of those formats back as a pd.DataFrame:
```
import statsmodels.api as sm

model = sm.OLS(y,x)
results = model.fit()
results_summary = results.summary()

# Note that tables is a list. The table at index 1 is the "core" table. Additionally, read_html puts dfs in a list, so we want index 0
results_as_html = results_summary.tables[1].as_html()
pd.read_html(results_as_html, header=0, index_col=0)[0]
```


Question:
Column of lists, convert list to string as a new column I have a dataframe with a column of lists which can be created with:
```
import pandas as pd
lists={1:[[1,2,12,6,'ABC']],2:[[1000,4,'z','a']]}
#create test dataframe
df=pd.DataFrame.from_dict(lists,orient='index')
df=df.rename(columns={0:'lists'})
```
The dataframe df looks like:
```
lists
1  [1, 2, 12, 6, ABC]
2     [1000, 4, z, a]
```
I need to create a new column called 'liststring' which takes every element of each list in lists and creates a string with each element separated by commas. The elements of each list can be int, float, or string. So the result would be:
```
lists    liststring
1  [1, 2, 12, 6, ABC]  1,2,12,6,ABC
2     [1000, 4, z, a]    1000,4,z,a
```
I have tried various things, including from How do I convert a list in a Pandas DF into a string?:
```
df['liststring']=df.lists.apply(lambda x: ', '.join(str(x)))
```
but unfortunately the result takes every character and seperates by comma:
```
lists                                         liststring
1  [1, 2, 12, 6, ABC]  [, 1, ,,  , 2, ,,  , 1, 2, ,,  , 6, ,,  , ', A...
2     [1000, 4, z, a]  [, 1, 0, 0, 0, ,,  , 4, ,,  , ', z, ', ,,  , '...
```

Response:
List Comprehension If performance is important, I strongly recommend this solution and I can explain why.
```
df['liststring'] = [','.join(map(str, l)) for l in df['lists']]
df

                lists    liststring
0  [1, 2, 12, 6, ABC]  1,2,12,6,ABC
1     [1000, 4, z, a]    1000,4,z,a
```
You can extend this to more complicated use cases using a function.
```
def try_join(l):
    try:
        return ','.join(map(str, l))
    except TypeError:
        return np.nan

df['liststring'] = [try_join(l) for l in df['lists']]
```
Series.apply/Series.agg with ','.join You need to convert your list items to strings first, that's where the map comes in handy.
```
df['liststring'] = df['lists'].apply(lambda x: ','.join(map(str, x)))
```
Or,
```
df['liststring'] = df['lists'].agg(lambda x: ','.join(map(str, x)))
df
                lists    liststring
0  [1, 2, 12, 6, ABC]  1,2,12,6,ABC
1     [1000, 4, z, a]    1000,4,z,a
```
pd.DataFrame constructor with DataFrame.agg A non-loopy/non-lambda solution.
```
df['liststring'] = (pd.DataFrame(df.lists.tolist())
                      .fillna('')
                      .astype(str)
                      .agg(','.join, 1)
                      .str.strip(','))

df
                lists    liststring
0  [1, 2, 12, 6, ABC]  1,2,12,6,ABC
1     [1000, 4, z, a]    1000,4,z,a
```


Question: Pandas fill missing values in dataframe from another dataframe I cannot find a pandas function (which I had seen before) to substitute the NaN's in a dataframe with values from another dataframe (assuming a common index which can be specified). Any help?

Response:
If you have two DataFrames of the same shape, then:
```
df[df.isnull()] = d2
```
Will do the trick. Only locations where df.isnull() evaluates to True (highlighted in green) will be eligible for assignment. In practice, the DataFrames aren't always the same size / shape, and transforming methods (especially .shift()) are useful. Data coming in is invariably dirty, incomplete, or inconsistent. Par for the course. There's a pretty extensive pandas tutorial and associated cookbook for dealing with these situations.


Question:
Pandas, groupby and count [duplicate] This question already has answers here: Pandas DataFrame Groupby two columns and get counts (8 answers) Closed 1 year ago. I have a dataframe say like this
```

```python
df = pd.DataFrame({'user_id':['a','a','s','s','s'],
#Output
#                    'session':[4,5,4,5,5],
#                    'revenue':[-1,0,1,2,1]})
```

```python
df
#Output
#   revenue  session user_id
#0       -1        4       a
#1        0        5       a
#2        1        4       s
#3        2        5       s
#4        1        5       s
#```
#And each value of session and revenue represents a kind of type, and I want to count the number of each kind say the number of revenue=-1 and session=4 of user_id=a is 1. And I found simple call count() function after groupby() can't output the result I want.
#```
```

```python
df.groupby('user_id').count()
#Output
#         revenue  session
#user_id
#a              2        2
#s              3        3
#```
#How can I do that?
```

Response:
pandas >= 1.1: df.value_counts is available! From pandas 1.1, this will be my recommended method for counting the number of rows in groups (i.e., the group size). To count the number of non-nan rows in a group for a specific column, check out the accepted answer. Old
```
df.groupby(['A', 'B']).size()   # df.groupby(['A', 'B'])['C'].count()
```
New [✓]
```
df.value_counts(subset=['A', 'B'])
```
Note that size and count are not identical, the former counts all rows per group, the latter counts non-null rows only. See this other answer of mine for more. Minimal Example
```
pd.__version__
# '1.1.0.dev0+2004.g8d10bfb6f'

df = pd.DataFrame({'num_legs': [2, 4, 4, 6],
                   'num_wings': [2, 0, 0, 0]},
                  index=['falcon', 'dog', 'cat', 'ant'])
df
        num_legs  num_wings
falcon         2          2
dog            4          0
cat            4          0
ant            6          0
```
```
df.value_counts(subset=['num_legs', 'num_wings'], sort=False)

num_legs  num_wings
2         2            1
4         0            2
6         0            1
dtype: int64
```
Compare this output with
```
df.groupby(['num_legs', 'num_wings'])['num_legs'].size()

num_legs  num_wings
2         2            1
4         0            2
6         0            1
Name: num_legs, dtype: int64
```
Performance It's also faster if you don't sort the result:
```
%timeit df.groupby(['num_legs', 'num_wings'])['num_legs'].count()
%timeit df.value_counts(subset=['num_legs', 'num_wings'], sort=False)

640 µs ± 28.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
568 µs ± 6.88 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```


Question:
Imputation of missing values for categories in pandas The question is how to fill NaNs with most frequent levels for category column in pandas dataframe? In R randomForest package there is na.roughfix option : A completed data matrix or data frame. For numeric variables, NAs are replaced with column medians. For factor variables, NAs are replaced with the most frequent levels (breaking ties at random). If object contains no NAs, it is returned unaltered. in Pandas for numeric variables I can fill NaN values with :
```
df = df.fillna(df.median())
```

Response:
You can use df = df.fillna(df['Label'].value_counts().index[0]) to fill NaNs with the most frequent value from one column. If you want to fill every column with its own most frequent value you can use df = df.apply(lambda x:x.fillna(x.value_counts().index[0])) UPDATE 2018-25-10 ⬇ Starting from 0.13.1 pandas includes mode method for Series and Dataframes. You can use it to fill missing values for each column (using its own most frequent value) like this
```
df = df.fillna(df.mode().iloc[0])
```


Question:
pandas "case insensitive" in a string or "case ignore" Here is the code that works for lowercase and returns only "apple":
```
df2 = df1['company_name'].str.contains("apple", na=False)
```
I need this to find "apple", "APPLE", "Apple", etc. Something like:
```
df2 = df1['company_name'].str.contains.caseignore("apple", na=False)
```
is there such a function anywhere?

Response:
Series.str.contains has a case parameter that is True by default. Set it to False to do a case insensitive match.
```
df2 = df1['company_name'].str.contains("apple", na=False, case=False)
```


Question:
Python pandas slice dataframe by multiple index ranges What is the pythonic way to slice a dataframe by more index ranges (eg. by 10:12 and 25:28)? I want this in a more elegant way:
```
df = pd.DataFrame({'a':range(10,100)})
df.iloc[[i for i in range(10,12)] + [i for i in range(25,28)]]
```
Result:
```
a
10  20
11  21
25  35
26  36
27  37
```
Something like this would be more elegant:
```
df.iloc[(10:12, 25:28)]
```

Response:
You can use numpy's r_ "slicing trick":
```
df = pd.DataFrame({'a':range(10,100)})
df.iloc[pd.np.r_[10:12, 25:28]]
```
NOTE: this now gives a warning The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead. To do that, you can import numpy as np and then slice the following way:
```
df.iloc[np.r_[10:12, 25:28]]
```
This gives:
```
a
10  20
11  21
25  35
26  36
27  37
```


Question:
Hide axis label only, not entire axis, in Pandas plot I can clear the text of the xlabel in a Pandas plot with:
```py
plt.xlabel("")
```
Instead, is it possible to hide the label? May be something like .xaxis.label.set_visible(False).

Response:
From the Pandas docs - The plot method on Series and DataFrame is just a simple wrapper around plt.plot(): This means that anything you can do with matplolib, you can do with a Pandas DataFrame plot. pyplot has an axis() method that lets you set axis properties. Calling plt.axis('off') before calling plt.show() will turn off both axes.
```
df.plot()
plt.axis('off')
plt.show()
plt.close()
```
To control a single axis, you need to set its properties via the plot's Axes. For the x axis - (pyplot.axes().get_xaxis().....)
```
df.plot()
ax1 = plt.axes()
x_axis = ax1.axes.get_xaxis()
x_axis.set_visible(False)
plt.show()
plt.close()
```
Similarly to control an axis label, get the label and turn it off.
```
df.plot()
ax1 = plt.axes()
x_axis = ax1.axes.get_xaxis()
x_axis.set_label_text('foo')
x_label = x_axis.get_label()
##print isinstance(x_label, matplotlib.artist.Artist)
x_label.set_visible(False)
plt.show()
plt.close()
```
You can also get to the x axis like this
```
ax1 = plt.axes()
x_axis = ax1.xaxis
x_axis.set_label_text('foo')
x_axis.label.set_visible(False)
```
Or this
```
ax1 = plt.axes()
ax1.xaxis.set_label_text('foo')
ax1.xaxis.label.set_visible(False)
```
DataFrame.plot returns a matplotlib.axes.Axes or numpy.ndarray of them so you can get it/them when you call it.
```
axs = df.plot()
```
.set_visible() is an Artist method. The axes and their labels are Artists so they have Artist methods/attributes as well as their own. There are many ways to customize your plots. Sometimes you can find the feature you want browsing the Gallery and Examples


Question:
Pandas OHLC aggregation on OHLC data I understand that OHLC re-sampling of time series data in Pandas, using one column of data, will work perfectly, for example on the following dataframe:
```
>>df
ctime       openbid
1443654000  1.11700
1443654060  1.11700
...

df['ctime']  = pd.to_datetime(df['ctime'], unit='s')
df           = df.set_index('ctime')
df.resample('1H',  how='ohlc', axis=0, fill_method='bfill')

>>>
                     open     high     low       close
ctime
2015-09-30 23:00:00  1.11700  1.11700  1.11687   1.11697
2015-09-30 24:00:00  1.11700  1.11712  1.11697   1.11697
...
```
But what do I do if the data is already in an OHLC format? From what I can gather the OHLC method of the API calculates an OHLC slice for every column, hence if my data is in the format:
```
ctime  openbid  highbid   lowbid  closebid
0       1443654000  1.11700  1.11700  1.11687   1.11697
1       1443654060  1.11700  1.11712  1.11697   1.11697
2       1443654120  1.11701  1.11708  1.11699   1.11708
```
When I try to re-sample I get an OHLC for each of the columns, like so:
```
openbid                             highbid           \
                        open     high      low    close     open     high
ctime
2015-09-30 23:00:00  1.11700  1.11700  1.11700  1.11700  1.11700  1.11712
2015-09-30 23:01:00  1.11701  1.11701  1.11701  1.11701  1.11708  1.11708
...
                                        lowbid                             \
                         low    close     open     high      low    close
ctime
2015-09-30 23:00:00  1.11700  1.11712  1.11687  1.11697  1.11687  1.11697
2015-09-30 23:01:00  1.11708  1.11708  1.11699  1.11699  1.11699  1.11699
...

                    closebid
                        open     high      low    close
ctime
2015-09-30 23:00:00  1.11697  1.11697  1.11697  1.11697
2015-09-30 23:01:00  1.11708  1.11708  1.11708  1.11708
```
Is there a quick(ish) workaround for this that someone is willing to share please, without me having to get knee-deep in pandas manual? Thanks. ps, there is this answer - Converting OHLC stock data into a different timeframe with python and pandas - but it was 4 years ago, so I am hoping there has been some progress.

Response:
This is similar to the answer you linked, but it a little cleaner, and faster, because it uses the optimized aggregations, rather than lambdas. Note that the resample(...).agg(...) syntax requires pandas version 0.18.0.
```

```python
df.resample('1H').agg({'openbid': 'first',
                                 'highbid': 'max',
                                 'lowbid': 'min',
                                 'closebid': 'last'})
#Output
#lowbid  highbid  closebid  openbid
#ctime
#2015-09-30 23:00:00  1.11687  1.11712   1.11708    1.117
#```
```


Question:
Pandas count null values in a groupby function
```
df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar', 'foo', 'bar', 'foo', 'foo'],
               'B' : ['one', 'one', 'two', 'three', 'two', 'two', 'one', 'three'],
               'C' : [np.nan, 'bla2', np.nan, 'bla3', np.nan, np.nan, np.nan, np.nan]})
```
Output:
```
A      B     C
0  foo    one   NaN
1  bar    one  bla2
2  foo    two   NaN
3  bar  three  bla3
4  foo    two   NaN
5  bar    two   NaN
6  foo    one   NaN
7  foo  three   NaN
```
I would like to use groupby in order to count the number of NaN's for the different combinations of foo. Expected Output (EDIT):
```
A      B     C    D
0  foo    one   NaN    2
1  bar    one  bla2    0
2  foo    two   NaN    2
3  bar  three  bla3    0
4  foo    two   NaN    2
5  bar    two   NaN    1
6  foo    one   NaN    2
7  foo  three   NaN    1
```
Currently I am trying this:
```
df['count']=df.groupby(['A'])['B'].isnull().transform('sum')
```
But this is not working... Thank You

Response:
I think you need groupby with sum of NaN values:
```
df2 = df.C.isnull().groupby([df['A'],df['B']]).sum().astype(int).reset_index(name='count')
print(df2)
     A      B  count
0  bar    one      0
1  bar  three      0
2  bar    two      1
3  foo    one      2
4  foo  three      1
5  foo    two      2
```
Notice that the .isnull() is on the original Dataframe column, not on the groupby()-object. The groupby() does not have .isnull() but if it would have it, it would be expected to give the same result as with .isnull() on the original DataFrame. If need filter first add boolean indexing:
```
df = df[df['A'] == 'foo']
df2 = df.C.isnull().groupby([df['A'],df['B']]).sum().astype(int)
print(df2)
A    B
foo  one      2
     three    1
     two      2
```
Or simpler:
```
df = df[df['A'] == 'foo']
df2 = df['B'].value_counts()
print(df2)
one      2
two      2
three    1
Name: B, dtype: int64
```
EDIT: Solution is very similar, only add transform:
```
df['D'] = df.C.isnull().groupby([df['A'],df['B']]).transform('sum').astype(int)
print(df)
     A      B     C  D
0  foo    one   NaN  2
1  bar    one  bla2  0
2  foo    two   NaN  2
3  bar  three  bla3  0
4  foo    two   NaN  2
5  bar    two   NaN  1
6  foo    one   NaN  2
7  foo  three   NaN  1
```
Similar solution:
```
df['D'] = df.C.isnull()
df['D'] = df.groupby(['A','B'])['D'].transform('sum').astype(int)
print(df)
     A      B     C  D
0  foo    one   NaN  2
1  bar    one  bla2  0
2  foo    two   NaN  2
3  bar  three  bla3  0
4  foo    two   NaN  2
5  bar    two   NaN  1
6  foo    one   NaN  2
7  foo  three   NaN  1
```


Question: Random Sample of a subset of a dataframe in Pandas I have a pandas DataFrame with 100,000 rows and want to split it into 100 sections with 1000 rows in each of them. How do I draw a random sample of certain size (e.g. 50 rows) of just one of the 100 sections? The df is already ordered such that the first 1000 rows are from the first section, next 1000 rows from another, and so on.

Response:
You can use the sample method*:
```

```python
df = pd.DataFrame([[1, 2], [3, 4], [5, 6], [7, 8]], columns=["A", "B"])
```

```python
df.sample(2)
#Output
#A  B
#0  1  2
#2  5  6
```

```python
df.sample(2)
#Output
#A  B
#3  7  8
#0  1  2
#```
#*On one of the section DataFrames. Note: If you have a larger sample size that the size of the DataFrame this will raise an error unless you sample with replacement.
#```
```

```python
df.sample(5)
ValueError: Cannot take a larger sample than population when 'replace=False'
```

```python
df.sample(5, replace=True)
#Output
#A  B
#0  1  2
#1  3  4
#2  5  6
#3  7  8
#1  3  4
#```
```


Question:
Count occurrences of each of certain words in pandas dataframe I want to count the number of occurrences of each of certain words in a data frame. I currently do it using str.contains:
```
a = df2[df2['col1'].str.contains("sample")].groupby('col2').size()
n = a.apply(lambda x: 1).sum()
```
Is there a method to match regular expression and get the count of occurrences? In my case I have a large dataframe and I want to match around 100 strings.

Response:
Update: Original answer counts those rows which contain a substring. To count all the occurrences of a substring you can use .str.count:
```

```python
df = pd.DataFrame(['hello', 'world', 'hehe'], columns=['words'])
```

```python
df.words.str.count("he|wo")
#Output
#0    1
#1    1
#2    2
#Name: words, dtype: int64
```

```python
df.words.str.count("he|wo").sum()
#Output
#4
#```
#The str.contains method accepts a regular expression:
#```
#Definition: df.words.str.contains(self, pat, case=True, flags=0, na=nan)
#Docstring:
#Check whether given pattern is contained in each string in the array
#
#Parameters
#----------
#pat : string
#    Character sequence or regular expression
#case : boolean, default True
#    If True, case sensitive
#flags : int, default 0 (no flags)
#    re module flags, e.g. re.IGNORECASE
#na : default NaN, fill value for missing values.
#```
#For example:
#```
```

```python
df = pd.DataFrame(['hello', 'world'], columns=['words'])
```

```python
df
#Output
#words
#0  hello
#1  world
```

```python
df.words.str.contains(r'[hw]')
#Output
#0    True
#1    True
#Name: words, dtype: bool
```

```python
df.words.str.contains(r'he|wo')
#Output
#0    True
#1    True
#Name: words, dtype: bool
#```
#To count the occurences you can just sum this boolean Series:
#```
```

```python
df.words.str.contains(r'he|wo').sum()
#Output
#2
```

```python
df.words.str.contains(r'he').sum()
#Output
#1
#```
```


Question:
pandas left join and update existing column I am new to pandas and can't seem to get this to work with merge function:
```

```python
left       >>> right
#Output
#   a  b   c       a  c   d
#0  1  4   9    0  1  7  13
#1  2  5  10    1  2  8  14
#2  3  6  11    2  3  9  15
#3  4  7  12
#```
#With a left join on column a, I would like to update common columns BY THE JOINED KEYS. Note last value in column c is from LEFT table since there is no match.
#```
```

```python
final
#Output
#   a  b   c   d
#0  1  4   7   13
#1  2  5   8   14
#2  3  6   9   15
#3  4  7   12  NAN
#```
#How should I do this with Pandas merge function? Thank you.
```

Response:
You can use merge() between left and right with how='left' on 'a' column.
```

```python
final = left.merge(right, on='a', how='left')
```

```python
final
#Output
#a  b  c_x  c_y   d
#0  1  4    9    7  13
#1  2  5   10    8  14
#2  3  6   11    9  15
#3  4  7   12  NaN NaN
#```
#Replace NaN value from c_y with c_x value
#```
```

```python
final['c'] = final['c_y'].fillna(final['c_x'])
```

```python
final
#Output
#a  b  c_x  c_y   d   c
#0  1  4    9    7  13   7
#1  2  5   10    8  14   8
#2  3  6   11    9  15   9
#3  4  7   12  NaN NaN  12
#```
#Drop unwanted columns, and you have the result
#```
```

```python
final.drop(['c_x', 'c_y'], axis=1)
#Output
#a  b   d   c
#0  1  4  13   7
#1  2  5  14   8
#2  3  6  15   9
#3  4  7 NaN  12
#```
```


Question:
how set column as date index? My data sets looks like:
```
Date        Value
1/1/1988    0.62
1/2/1988    0.64
1/3/1988    0.65
1/4/1988    0.66
1/5/1988    0.67
1/6/1988    0.66
1/7/1988    0.64
1/8/1988    0.66
1/9/1988    0.65
1/10/1988   0.65
1/11/1988   0.64
1/12/1988   0.66
1/13/1988   0.67
1/14/1988   0.66
1/15/1988   0.65
1/16/1988   0.64
1/17/1988   0.62
1/18/1988   0.64
1/19/1988   0.62
1/20/1988   0.62
1/21/1988   0.64
1/22/1988   0.62
1/23/1988   0.60
```
I used this code to read this data:
```
df.set_index(df['Date'], drop=False, append=False, inplace=False, verify_integrity=False).drop('Date', 1)
```
But the problem is the index is not in date format. So the question is how to set this column as date index?

Response:
Your question lacked a proper explanation, but you can do the following:
```

```python
# convert to datetime
df['Date'] = pd.to_datetime(df['Date'])
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 23 entries, 0 to 22
Data columns (total 2 columns):
Date     23 non-null datetime64[ns]
Value    23 non-null float64
dtypes: datetime64[ns](1), float64(1)
memory usage: 448.0 bytes
```

```python
# set the index
df.set_index('Date', inplace=True)
df.info()

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 23 entries, 1988-01-01 to 1988-01-23
Data columns (total 1 columns):
Value    23 non-null float64
dtypes: float64(1)
memory usage: 368.0 bytes
```
So here to_datetime will convert date strings to datetime dtype, set_index with param inplace=True is all you need,
```


Question:
Pandas KeyError: value not in index I have the following code,
```
df = pd.read_csv(CsvFileName)

p = df.pivot_table(index=['Hour'], columns='DOW', values='Changes', aggfunc=np.mean).round(0)
p.fillna(0, inplace=True)

p[["1Sun", "2Mon", "3Tue", "4Wed", "5Thu", "6Fri", "7Sat"]] = p[["1Sun", "2Mon", "3Tue", "4Wed", "5Thu", "6Fri", "7Sat"]].astype(int)
```
It has always been working until the csv file doesn't have enough coverage (of all week days). For e.g., with the following .csv file,
```
DOW,Hour,Changes
4Wed,01,237
3Tue,07,2533
1Sun,01,240
3Tue,12,4407
1Sun,09,2204
1Sun,01,240
1Sun,01,241
1Sun,01,241
3Tue,11,662
4Wed,01,4
2Mon,18,4737
1Sun,15,240
2Mon,02,4
6Fri,01,1
1Sun,01,240
2Mon,19,2300
2Mon,19,2532
```
I'll get the following error:
```
KeyError: "['5Thu' '7Sat'] not in index"
```
It seems to have a very easy fix, but I'm just too new to Python to know how to fix it.

Response:
Use reindex to get all columns you need. It'll preserve the ones that are already there and put in empty columns otherwise.
```
p = p.reindex(columns=['1Sun', '2Mon', '3Tue', '4Wed', '5Thu', '6Fri', '7Sat'])
```
So, your entire code example should look like this:
```
df = pd.read_csv(CsvFileName)

p = df.pivot_table(index=['Hour'], columns='DOW', values='Changes', aggfunc=np.mean).round(0)
p.fillna(0, inplace=True)

columns = ["1Sun", "2Mon", "3Tue", "4Wed", "5Thu", "6Fri", "7Sat"]
p = p.reindex(columns=columns)
p[columns] = p[columns].astype(int)
```


Question: Export Pandas DataFrame into a PDF file using Python What is an efficient way to generate PDF for data frames in Pandas?

Response:
First plot table with matplotlib then generate pdf
```
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.backends.backend_pdf import PdfPages

df = pd.DataFrame(np.random.random((10,3)), columns = ("col 1", "col 2", "col 3"))

#https://stackoverflow.com/questions/32137396/how-do-i-plot-only-a-table-in-matplotlib
fig, ax =plt.subplots(figsize=(12,4))
ax.axis('tight')
ax.axis('off')
the_table = ax.table(cellText=df.values,colLabels=df.columns,loc='center')

#https://stackoverflow.com/questions/4042192/reduce-left-and-right-margins-in-matplotlib-plot
pp = PdfPages("foo.pdf")
pp.savefig(fig, bbox_inches='tight')
pp.close()
```
reference: How do I plot only a table in Matplotlib? Reduce left and right margins in matplotlib plot


Question:
How to resample a dataframe with different functions applied to each column? I have a times series with temperature and radiation in a pandas dataframe. The time resolution is 1 minute in regular steps.
```
import datetime
import pandas as pd
import numpy as np

date_times = pd.date_range(datetime.datetime(2012, 4, 5, 8, 0),
                           datetime.datetime(2012, 4, 5, 12, 0),
                           freq='1min')
tamb = np.random.sample(date_times.size) * 10.0
radiation = np.random.sample(date_times.size) * 10.0
frame = pd.DataFrame(data={'tamb': tamb, 'radiation': radiation},
                     index=date_times)
frame
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 241 entries, 2012-04-05 08:00:00 to 2012-04-05 12:00:00
Freq: T
Data columns:
radiation    241  non-null values
tamb         241  non-null values
dtypes: float64(2)
```
How can I down-sample this dataframe to a resolution of one hour, computing the hourly mean for the temperature and the hourly sum for radiation?

Response:
With pandas 0.18 the resample API changed (see the docs). So for pandas >= 0.18 the answer is:
```

```python
frame.resample('1H').agg({'radiation': np.sum, 'tamb': np.mean})
#Output
#tamb   radiation
#2012-04-05 08:00:00  5.161235  279.507182
#2012-04-05 09:00:00  4.968145  290.941073
#2012-04-05 10:00:00  4.478531  317.678285
#2012-04-05 11:00:00  4.706206  335.258633
#2012-04-05 12:00:00  2.457873    8.655838
#```
#Old Answer: I am answering my question to reflect the time series related changes in pandas >= 0.8 (all other answers are outdated). Using pandas >= 0.8 the answer is:
#```
```

```python
frame.resample('1H', how={'radiation': np.sum, 'tamb': np.mean})
#Output
#tamb   radiation
#2012-04-05 08:00:00  5.161235  279.507182
#2012-04-05 09:00:00  4.968145  290.941073
#2012-04-05 10:00:00  4.478531  317.678285
#2012-04-05 11:00:00  4.706206  335.258633
#2012-04-05 12:00:00  2.457873    8.655838
#```
```


Question:
Merge two python pandas data frames of different length but keep all rows in output data frame I have the following problem: I have two pandas data frames of different length containing some rows and columns that have common values and some that are different, like this:
```
df1:                                 df2:

      Column1  Column2  Column3           ColumnA  ColumnB ColumnC
    0    a        x        x            0    c        y       y
    1    c        x        x            1    e        z       z
    2    e        x        x            2    a        s       s
    3    d        x        x            3    d        f       f
    4    h        x        x
    5    k        x        x
```
What I want to do now is merging the two dataframes so that if ColumnA and Column1 have the same value the rows from df2 are appended to the corresponding row in df1, like this:
```
df1:
    Column1  Column2  Column3  ColumnB  ColumnC
  0    a        x        x        s        s
  1    c        x        x        y        y
  2    e        x        x        z        z
  3    d        x        x        f        f
  4    h        x        x        NaN      NaN
  5    k        x        x        NaN      NaN
```
I know that the merge is doable through
```
df1.merge(df2,left_on='Column1', right_on='ColumnA')
```
but this command drops all rows that are not the same in Column1 and ColumnA in both files. Instead of that I want to keep these rows in df1 and just assign NaN to them in the columns where other rows have a value from df2, as shown above. Is there a smooth way to do this in pandas?

Response:
You can read the documentation here: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html What you are looking for is a left join. The default option is an inner join. You can change this behavior by passing a different how argument:
```
df1.merge(df2,how='left', left_on='Column1', right_on='ColumnA')
```


Question:
Missing data, insert rows in Pandas and fill with NAN I'm new to Python and Pandas so there might be a simple solution which I don't see. I have a number of discontinuous datasets which look like this:
```
ind A    B  C
0   0.0  1  3
1   0.5  4  2
2   1.0  6  1
3   3.5  2  0
4   4.0  4  5
5   4.5  3  3
```
I now look for a solution to get the following:
```
ind A    B  C
0   0.0  1  3
1   0.5  4  2
2   1.0  6  1
3   1.5  NAN NAN
4   2.0  NAN NAN
5   2.5  NAN NAN
6   3.0  NAN NAN
7   3.5  2  0
8   4.0  4  5
9   4.5  3  3
```
The problem is,that the gap in A varies from dataset to dataset in position and length...

Response:
set_index and reset_index are your friends.
```
df = DataFrame({"A":[0,0.5,1.0,3.5,4.0,4.5], "B":[1,4,6,2,4,3], "C":[3,2,1,0,5,3]})
```
First move column A to the index:
```

```python
df.set_index("A")
#Output
#B  C
# A
#0.0  1  3
#0.5  4  2
#1.0  6  1
#3.5  2  0
#4.0  4  5
#4.5  3  3
#```
#Then reindex with a new index, here the missing data is filled in with nans. We use the Index object since we can name it; this will be used in the next step.
#```
```

```python
new_index = Index(arange(0,5,0.5), name="A")
```

```python
df.set_index("A").reindex(new_index)
#Output
#B   C
#0.0   1   3
#0.5   4   2
#1.0   6   1
#1.5 NaN NaN
#2.0 NaN NaN
#2.5 NaN NaN
#3.0 NaN NaN
#3.5   2   0
#4.0   4   5
#4.5   3   3
#```
#Finally move the index back to the columns with reset_index. Since we named the index, it all works magically:
#```
```

```python
df.set_index("A").reindex(new_index).reset_index()
#Output
#A   B   C
#0    0.0   1   3
#1    0.5   4   2
#2    1.0   6   1
#3    1.5 NaN NaN
#4    2.0 NaN NaN
#5    2.5 NaN NaN
#6    3.0 NaN NaN
#7    3.5   2   0
#8    4.0   4   5
#9    4.5   3   3
#```
```


Question:
Number of rows changes even after `pandas.merge` with `left` option I am merging two data frames using pandas.merge. Even after specifying how = left option, I found the number of rows of merged data frame is larger than the original. Why does this happen?
```
panel = pd.read_csv(file1, encoding ='cp932')
before_len = len(panel)

prof_2000 = pd.read_csv(file2, encoding ='cp932').drop_duplicates()

temp_2000 = pd.merge(panel, prof_2000, left_on='Candidate_u', right_on="name2", how="left")

after_len =  len(temp_2000)
print(before_len, after_len)
> 12661 13915
```

Response:
This sounds like having more than one rows in right under 'name2' that match the key you have set for the left. Using option 'how='left' with pandas.DataFrame.merge() only means that: left: use only keys from left frame However, the actual number of rows in the result object is not necessarily going to be the same as the number of rows in the left object. Example:
```

```python
df_1
#Output
#A    B
#0  a  AAA
#1  b  BBA
#2  c  CCF
#```
#and then another DF that looks like this (notice that there are more than one entry for your desired key on the left):
#```
```

```python
df_3
#Output
#key  value
#0   a      1
#1   a      2
#2   b      3
#3   a      4
#```
#If I merge these two on left.A, here's what happens:
#```
```

```python
df_1.merge(df_3, how='left', left_on='A', right_on='key')
#Output
#A    B  key  value
#0  a  AAA    a    1.0
#1  a  AAA    a    2.0
#2  a  AAA    a    4.0
#3  b  BBA    b    3.0
#4  c  CCF  NaN    NaN
#```
#This happened even though I merged with how='left' as you can see above, there were simply more than one rows to merge and as shown here the result pd.DataFrame has in fact more rows than the pd.DataFrame on the left. I hope this helps!
```


Question:
Multiple histograms in Pandas I would like to create the following histogram (see image below) taken from the book "Think Stats". However, I cannot get them on the same plot. Each DataFrame takes its own subplot. I have the following code:
```
import nsfg
import matplotlib.pyplot as plt
df = nsfg.ReadFemPreg()
preg = nsfg.ReadFemPreg()
live = preg[preg.outcome == 1]

first = live[live.birthord == 1]
others = live[live.birthord != 1]

#fig = plt.figure()
#ax1 = fig.add_subplot(111)

first.hist(column = 'prglngth', bins = 40, color = 'teal', \
           alpha = 0.5)
others.hist(column = 'prglngth', bins = 40, color = 'blue', \
            alpha = 0.5)
plt.show()
```
The above code does not work when I use ax = ax1 as suggested in: pandas multiple plots not working as hists nor this example does what I need: Overlaying multiple histograms using pandas. When I use the code as it is, it creates two windows with histograms. Any ideas how to combine them? Here's an example of how I'd like the final figure to look:

Response:
As far as I can tell, pandas can't handle this situation. That's ok since all of their plotting methods are for convenience only. You'll need to use matplotlib directly. Here's how I do it:
```
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import pandas
#import seaborn
#seaborn.set(style='ticks')

np.random.seed(0)
df = pandas.DataFrame(np.random.normal(size=(37,2)), columns=['A', 'B'])
fig, ax = plt.subplots()

a_heights, a_bins = np.histogram(df['A'])
b_heights, b_bins = np.histogram(df['B'], bins=a_bins)

width = (a_bins[1] - a_bins[0])/3

ax.bar(a_bins[:-1], a_heights, width=width, facecolor='cornflowerblue')
ax.bar(b_bins[:-1]+width, b_heights, width=width, facecolor='seagreen')
#seaborn.despine(ax=ax, offset=10)
```
And that gives me:


Question:
Boolean Indexing with multiple conditions [duplicate] This question already has answers here: Logical operators for Boolean indexing in Pandas (4 answers) Closed 6 years ago. I have a Pandas DF where I need to filter out some rows that contains values == 0 for feature 'a' and feature 'b'. In order to inspect the values, I run the following:
```
DF1 = DF[DF['a'] == 0]
```
Which returns the right values. Similarly, by doing this:
```
DF2 = DF[DF['b'] == 0]
```
I can see the 0 values for feature 'b'. However, if I try to combine these 2 in a single line of code using the OR operand:
```
DF3 = DF[DF['a'] == 0 |  DF['b'] == 0]
```
I get this:
```
TypeError: cannot compare a dtyped [float64] array with a scalar of type [bool]
```
What's happening here?

Response:
You can transform either column 'a' or 'b' so they are both either float64 or bool. However, an easier solution that preserves the data type of your features is this:
```
DF3 = DF[(DF['a'] == 0) | (DF['b'] == 0)]
```
A common operation is the use of boolean vectors to filter the data. The operators are: | for or, & for and, and ~ for not. These must be grouped by using parentheses.


Question: What are Pandas "expanding window" functions? Pandas documentation lists a bunch of "expanding window functions" : http://pandas.pydata.org/pandas-docs/version/0.17.0/api.html#standard-expanding-window-functions But I couldn't figure out what they do from the documentation. Are they equivalent to the rolling functions, but dropping the NA values?

Response:
You may want to read this Pandas docs: A common alternative to rolling statistics is to use an expanding window, which yields the value of the statistic with all the data available up to that point in time. These follow a similar interface to .rolling, with the .expanding method returning an Expanding object. As these calculations are a special case of rolling statistics, they are implemented in pandas such that the following two calls are equivalent:
```

```python
df.rolling(window=len(df), min_periods=1).mean()[:5]
#Output
#A         B         C         D
#2000-01-01  0.314226 -0.001675  0.071823  0.892566
#2000-01-02  0.654522 -0.171495  0.179278  0.853361
#2000-01-03  0.708733 -0.064489 -0.238271  1.371111
#2000-01-04  0.987613  0.163472 -0.919693  1.566485
#2000-01-05  1.426971  0.288267 -1.358877  1.808650
```

```python
df.expanding(min_periods=1).mean()[:5]
#Output
#A         B         C         D
#2000-01-01  0.314226 -0.001675  0.071823  0.892566
#2000-01-02  0.654522 -0.171495  0.179278  0.853361
#2000-01-03  0.708733 -0.064489 -0.238271  1.371111
#2000-01-04  0.987613  0.163472 -0.919693  1.566485
#2000-01-05  1.426971  0.288267 -1.358877  1.808650
#```
```


Question:
pandas, melt, unmelt preserve index I've got a table of clients (coper) and asset allocation (asset)
```
A = [[1,2],[3,4],[5,6]]
idx = ['coper1','coper2','coper3']
cols = ['asset1','asset2']

df = pd.DataFrame(A,index = idx, columns = cols)
```
so my data look like
```
asset1  asset2
coper1       1       2
coper2       3       4
coper3       5       6
```
and I want to run them through a linear optimization (i've got constraints- somtehing like sum of all of asset_i <= amount_on_hand_i and sum of coper_j = price_j) so I have to turn this 2D matrix into a 1D vector. Which is easy with melt
```
df2 = pd.melt(df,value_vars=['asset1','asset2'])
```
But now, when I try to unmelt it, I get a 6-row array with lots of blanks!
```
df2.pivot(columns = 'variable', values = 'value')

variable  asset1  asset2
0            1.0     NaN
1            3.0     NaN
2            5.0     NaN
3            NaN     2.0
4            NaN     4.0
5            NaN     6.0
```
Is there any way to preserve the 'coper' part of my indexing while using melt?

Response:
You need preserve index values by reset_index and parameter id_vars:
```
df2 = pd.melt(df.reset_index(), id_vars='index',value_vars=['asset1','asset2'])
print (df2)
    index variable  value
0  coper1   asset1      1
1  coper2   asset1      3
2  coper3   asset1      5
3  coper1   asset2      2
4  coper2   asset2      4
5  coper3   asset2      6
```
Then pivot working nice:
```
print(df2.pivot(index='index',columns = 'variable', values = 'value'))
variable  asset1  asset2
index
coper1         1       2
coper2         3       4
coper3         5       6
```
Another possible solution with stack:
```
df2 = df.stack().reset_index()
df2.columns = list('abc')
print (df2)
        a       b  c
0  coper1  asset1  1
1  coper1  asset2  2
2  coper2  asset1  3
3  coper2  asset2  4
4  coper3  asset1  5
5  coper3  asset2  6

print(df2.pivot(index='a',columns = 'b', values = 'c'))
b       asset1  asset2
a
coper1       1       2
coper2       3       4
coper3       5       6
```


Question:
Pandas filling missing dates and values within group I've a data frame that looks like the following
```
x = pd.DataFrame({'user': ['a','a','b','b'], 'dt': ['2016-01-01','2016-01-02', '2016-01-05','2016-01-06'], 'val': [1,33,2,1]})
```
What I would like to be able to do is find the minimum and maximum date within the date column and expand that column to have all the dates there while simultaneously filling in 0 for the val column. So the desired output is
```
dt user  val
0   2016-01-01    a    1
1   2016-01-02    a   33
2   2016-01-03    a    0
3   2016-01-04    a    0
4   2016-01-05    a    0
5   2016-01-06    a    0
6   2016-01-01    b    0
7   2016-01-02    b    0
8   2016-01-03    b    0
9   2016-01-04    b    0
10  2016-01-05    b    2
11  2016-01-06    b    1
```
I've tried the solution mentioned here and here but they aren't what I'm after. Any pointers much appreciated.

Response:
Initial Dataframe:
```
dt  user    val
0   2016-01-01     a      1
1   2016-01-02     a     33
2   2016-01-05     b      2
3   2016-01-06     b      1
```
First, convert the dates to datetime:
```
x['dt'] = pd.to_datetime(x['dt'])
```
Then, generate the dates and unique users:
```
dates = x.set_index('dt').resample('D').asfreq().index

>> DatetimeIndex(['2016-01-01', '2016-01-02', '2016-01-03', '2016-01-04',
               '2016-01-05', '2016-01-06'],
              dtype='datetime64[ns]', name='dt', freq='D')

users = x['user'].unique()

>> array(['a', 'b'], dtype=object)
```
This will allow you to create a MultiIndex:
```
idx = pd.MultiIndex.from_product((dates, users), names=['dt', 'user'])

>> MultiIndex(levels=[[2016-01-01 00:00:00, 2016-01-02 00:00:00, 2016-01-03 00:00:00, 2016-01-04 00:00:00, 2016-01-05 00:00:00, 2016-01-06 00:00:00], ['a', 'b']],
           labels=[[0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5], [0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1]],
           names=['dt', 'user'])
```
You can use that to reindex your DataFrame:
```
x.set_index(['dt', 'user']).reindex(idx, fill_value=0).reset_index()
Out:
           dt user  val
0  2016-01-01    a    1
1  2016-01-01    b    0
2  2016-01-02    a   33
3  2016-01-02    b    0
4  2016-01-03    a    0
5  2016-01-03    b    0
6  2016-01-04    a    0
7  2016-01-04    b    0
8  2016-01-05    a    0
9  2016-01-05    b    2
10 2016-01-06    a    0
11 2016-01-06    b    1
```
which then can be sorted by users:
```
x.set_index(['dt', 'user']).reindex(idx, fill_value=0).reset_index().sort_values(by='user')
Out:
           dt user  val
0  2016-01-01    a    1
2  2016-01-02    a   33
4  2016-01-03    a    0
6  2016-01-04    a    0
8  2016-01-05    a    0
10 2016-01-06    a    0
1  2016-01-01    b    0
3  2016-01-02    b    0
5  2016-01-03    b    0
7  2016-01-04    b    0
9  2016-01-05    b    2
11 2016-01-06    b    1
```


Question:
Best way to count the number of rows with missing values in a pandas DataFrame I currently came up with some work arounds to count the number of missing values in a pandas DataFrame. Those are quite ugly and I am wondering if there is a better way to do it. Let's create an example DataFrame:
```
from numpy.random import randn
df = pd.DataFrame(randn(5, 3), index=['a', 'c', 'e', 'f', 'h'],
               columns=['one', 'two', 'three'])
df = df.reindex(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'])
```
What I currently have is a) Counting cells with missing values:
```

```python
sum(df.isnull().values.ravel())
#Output
#9
#```
#b) Counting rows that have missing values somewhere:
#```
```

```python
sum([True for idx,row in df.iterrows() if any(row.isnull())])
#Output
#3
#```
```

Response:
For the second count I think just subtract the number of rows from the number of rows returned from dropna:
```

```python
from numpy.random import randn
df = pd.DataFrame(randn(5, 3), index=['a', 'c', 'e', 'f', 'h'],
               columns=['one', 'two', 'three'])
df = df.reindex(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'])
df
#Output
#one       two     three
#a -0.209453 -0.881878  3.146375
#b       NaN       NaN       NaN
#c  0.049383 -0.698410 -0.482013
#d       NaN       NaN       NaN
#e -0.140198 -1.285411  0.547451
#f -0.219877  0.022055 -2.116037
#g       NaN       NaN       NaN
#h -0.224695 -0.025628 -0.703680
```

```python
df.shape[0] - df.dropna().shape[0]
#Output
#3
#```
#The first could be achieved using the built in methods:
#```
```

```python
df.isnull().values.ravel().sum()
#Output
#9
#```
#Timings
#```
```

```python
%timeit sum([True for idx,row in df.iterrows() if any(row.isnull())])
%timeit df.shape[0] - df.dropna().shape[0]
%timeit sum(map(any, df.apply(pd.isnull)))
1000 loops, best of 3: 1.55 ms per loop
1000 loops, best of 3: 1.11 ms per loop
1000 loops, best of 3: 1.82 ms per loop
```

```python
%timeit sum(df.isnull().values.ravel())
%timeit df.isnull().values.ravel().sum()
%timeit df.isnull().sum().sum()
1000 loops, best of 3: 215 µs per loop
1000 loops, best of 3: 210 µs per loop
1000 loops, best of 3: 605 µs per loop
```
So my alternatives are a little faster for a df of this size Update So for a df with 80,000 rows I get the following:
```
```

```python
%timeit sum([True for idx,row in df.iterrows() if any(row.isnull())])
%timeit df.shape[0] - df.dropna().shape[0]
%timeit sum(map(any, df.apply(pd.isnull)))
%timeit np.count_nonzero(df.isnull())
1 loops, best of 3: 9.33 s per loop
100 loops, best of 3: 6.61 ms per loop
100 loops, best of 3: 3.84 ms per loop
1000 loops, best of 3: 395 µs per loop
```

```python
%timeit sum(df.isnull().values.ravel())
%timeit df.isnull().values.ravel().sum()
%timeit df.isnull().sum().sum()
%timeit np.count_nonzero(df.isnull().values.ravel())
1000 loops, best of 3: 675 µs per loop
1000 loops, best of 3: 679 µs per loop
100 loops, best of 3: 6.56 ms per loop
1000 loops, best of 3: 368 µs per loop
```
Actually np.count_nonzero wins this hands down.
```


Question:
Insert a link inside a Pandas table I'd like to insert a link (to a web page) inside a Pandas table, so when it is displayed in an IPython notebook, I could press the link. I tried the following:
```

```python
import pandas as pd
```

```python
df = pd.DataFrame(range(5), columns=['a'])
```

```python
df['b'] = df['a'].apply(lambda x: 'http://example.com/{0}'.format(x))
```

```python
df
#Output
#a                     b
#0  0  http://example.com/0
#1  1  http://example.com/1
#2  2  http://example.com/2
#3  3  http://example.com/3
#4  4  http://example.com/4
#```
#But the URL is just displayed as text. I also tried using an IPython HTML object:
#```
```

```python
from IPython.display import HTML
```

```python
df['b'] = df['a'].apply(lambda x:HTML('http://example.com/{0}'.format(x)))
```

```python
df
#Output
#a                                                 b
#0  0  <IPython.core.display.HTML object at 0x0481E530>
#1  1  <IPython.core.display.HTML object at 0x0481E770>
#2  2  <IPython.core.display.HTML object at 0x0481E7B0>
#3  3  <IPython.core.display.HTML object at 0x0481E810>
#4  4  <IPython.core.display.HTML object at 0x0481EA70>
#```
#But it will only display the repr of the object. Any other ideas? alko got the right answer. I just wanted to add that the cell width is limited by default, and long HTML code will be truncated, i.e.:
#```
#xxx
#```
#will become this:
#```
#<a href="aaaaaaaaaaaaaaaaaaaaaa...
#```
#and won't be displayed correctly. (Even though the text xxx is short and can fit in the cell.) I've bypassed it by setting:
#```
#pd.set_printoptions(max_colwidth=-1)
#```
```

Response:
I suppose you have to represent whole Pandas object as an HTML object, that is
```

```python
from IPython.display import HTML
```

```python
df = pd.DataFrame(list(range(5)), columns=['a'])
```

```python
df['a'] = df['a'].apply(lambda x: 'link'.format(x))
```

```python
HTML(df.to_html(escape=False))
```
Sorry, now I don't have IPython at hand, and can't check whether the output is correct.
```


Question:
Adding a column in pandas df using a function I have a Pandas df [see below]. How do I add values from a function to a new column "price"? function:
```
def getquotetoday(symbol):
    yahoo = Share(symbol)
    return yahoo.get_prev_close()
```
df:
```
Symbol      Bid     Ask
MSFT      10.25   11.15
AAPL     100.01  102.54
...
```

Response:
In general, you can use the apply function. If your function requires only one column, you can use:
```
df['price'] = df['Symbol'].apply(getquotetoday)
```
as @EdChum suggested. If your function requires multiple columns, you can use something like:
```
df['new_column_name'] = df.apply(lambda x: my_function(x['value_1'], x['value_2']), axis=1)
```


Question:
python pandas, DF.groupby().agg(), column reference in agg() On a concrete problem, say I have a DataFrame DF
```
word  tag count
0    a     S    30
1    the   S    20
2    a     T    60
3    an    T    5
4    the   T    10
```
I want to find, for every "word", the "tag" that has the most "count". So the return would be something like
```
word  tag count
1    the   S    20
2    a     T    60
3    an    T    5
```
I don't care about the count column or if the order/Index is original or messed up. Returning a dictionary {'the' : 'S', ...} is just fine. I hope I can do
```
DF.groupby(['word']).agg(lambda x: x['tag'][ x['count'].argmax() ] )
```
but it doesn't work. I can't access column information. More abstractly, what does the function in agg(function) see as its argument? btw, is .agg() the same as .aggregate() ? Many thanks.

Response:
agg is the same as aggregate. It's callable is passed the columns (Series objects) of the DataFrame, one at a time. You could use idxmax to collect the index labels of the rows with the maximum count:
```
idx = df.groupby('word')['count'].idxmax()
print(idx)
```
yields
```
word
a       2
an      3
the     1
Name: count
```
and then use loc to select those rows in the word and tag columns:
```
print(df.loc[idx, ['word', 'tag']])
```
yields
```
word tag
2    a   T
3   an   T
1  the   S
```
Note that idxmax returns index labels. df.loc can be used to select rows by label. But if the index is not unique -- that is, if there are rows with duplicate index labels -- then df.loc will select all rows with the labels listed in idx. So be careful that df.index.is_unique is True if you use idxmax with df.loc Alternative, you could use apply. apply's callable is passed a sub-DataFrame which gives you access to all the columns:
```
import pandas as pd
df = pd.DataFrame({'word':'a the a an the'.split(),
                   'tag': list('SSTTT'),
                   'count': [30, 20, 60, 5, 10]})

print(df.groupby('word').apply(lambda subf: subf['tag'][subf['count'].idxmax()]))
```
yields
```
word
a       T
an      T
the     S
```
Using idxmax and loc is typically faster than apply, especially for large DataFrames. Using IPython's %timeit:
```
N = 10000
df = pd.DataFrame({'word':'a the a an the'.split()*N,
                   'tag': list('SSTTT')*N,
                   'count': [30, 20, 60, 5, 10]*N})
def using_apply(df):
    return (df.groupby('word').apply(lambda subf: subf['tag'][subf['count'].idxmax()]))

def using_idxmax_loc(df):
    idx = df.groupby('word')['count'].idxmax()
    return df.loc[idx, ['word', 'tag']]

```python
%timeit using_apply(df)
100 loops, best of 3: 7.68 ms per loop
```

```python
%timeit using_idxmax_loc(df)
100 loops, best of 3: 5.43 ms per loop
```
If you want a dictionary mapping words to tags, then you could use set_index and to_dict like this:
```
```

```python
df2 = df.loc[idx, ['word', 'tag']].set_index('word')
```

```python
df2
#Output
#tag
#word
#a      T
#an     T
#the    S
```

```python
df2.to_dict()['tag']
#Output
#{'a': 'T', 'an': 'T', 'the': 'S'}
#```
```


Question:
How to get a single value as a string from pandas dataframe I am querying a single value from my data frame which seems to be 'dtype: object'. I simply want to print the value as it is with out printing the index or other information as well. How do I do this?
```
col_names = ['Host', 'Port']
df = pd.DataFrame(columns=col_names)
df.loc[len(df)] = ['a', 'b']

t = df[df['Host'] == 'a']['Port']
print(t)
```
OUTPUT: EXPECTED OUTPUT: b

Response:
If you can guarantee only one result is returned, use loc and call item:
```

```python
df.loc[df['Host'] == 'a', 'Port'].item()
#Output
#'b'
#```
#Or, similarly,
#```
```

```python
df.loc[df['Host'] == 'a', 'Port'].values[0]
#Output
#'b'
#```
#...to get the first value (similarly, .values[1] for the second). Which is better than df.loc[df['Host'] == 'a', 'Port'][0] because, if your DataFrame looks like this,
#```
#Host Port
#1    a    b
#```
#Then "KeyError: 0" will be thrown—
#```
#df.loc[df['Host'] == 'a', 'Port'][0]
#---------------------------------------------------------------------------
#KeyError                                  Traceback (most recent call last)
#```
#Alternatively, use at:
#```
```

```python
df.at[df['Host'].eq('a').idxmax(), 'Port']
#Output
#'b'
#```
#The drawback is that if 'a' doesn't exist, idxmax will return the first index (and return an incorrect result).
```


Question:
How to create a stacked bar chart for my DataFrame using seaborn [duplicate] This question already has answers here: How to have clusters of stacked bars (10 answers) Closed 7 years ago. I have a DataFrame df:
```py
df = pd.DataFrame(columns=["App","Feature1", "Feature2","Feature3", "Feature4","Feature5", "Feature6","Feature7","Feature8"], data=[['SHA', 0, 0, 1, 1, 1, 0, 1, 0], ['LHA', 1, 0, 1, 1, 0, 1, 1, 0], ['DRA', 0, 0, 0, 0, 0, 0, 1, 0], ['FRA', 1, 0, 1, 1, 1, 0, 1, 1], ['BRU', 0, 0, 1, 0, 1, 0, 0, 0], ['PAR', 0, 1, 1, 1, 1, 0, 1, 0], ['AER', 0, 0, 1, 1, 0, 1, 1, 0], ['SHE', 0, 0, 0, 1, 0, 0, 1, 0]])

# display(df)
   App  Feature1  Feature2  Feature3  Feature4  Feature5  Feature6  Feature7  Feature8
0  SHA         0         0         1         1         1         0         1         0
1  LHA         1         0         1         1         0         1         1         0
2  DRA         0         0         0         0         0         0         1         0
3  FRA         1         0         1         1         1         0         1         1
4  BRU         0         0         1         0         1         0         0         0
5  PAR         0         1         1         1         1         0         1         0
6  AER         0         0         1         1         0         1         1         0
7  SHE         0         0         0         1         0         0         1         0
```
I want to create a stacked bar chart so that each stack would correspond to App while the Y axis would contain the count of 1 values and the X axis would be Feature. It should be similar to this bar chart with the only difference that now I want to see stack bars and a legend with colors:
```py
df_c = df.iloc[:, 1:].eq(1).sum().rename_axis('Feature').reset_index(name='Count')
df_c = df_c.sort_values('Count')
plt.figure(figsize=(12,8))
ax = sns.barplot(x="Feature", y='Count', data=df_c, palette=sns.color_palette("GnBu", 10))
plt.xticks(rotation='vertical')
ax.grid(b=True, which='major', color='#d3d3d3', linewidth=1.0)
ax.grid(b=True, which='minor', color='#d3d3d3', linewidth=0.5)
plt.show()
```

Response:
You could use pandas plot as @Bharath suggest:
```
import seaborn as sns
sns.set()
df.set_index('App').T.plot(kind='bar', stacked=True)
```
Output: Updated: from matplotlib.colors import ListedColormap df.set_index('App')\ .reindex_axis(df.set_index('App').sum().sort_values().index, axis=1)\ .T.plot(kind='bar', stacked=True, colormap=ListedColormap(sns.color_palette("GnBu", 10)), figsize=(12,6)) Updated Pandas 0.21.0+ reindex_axis is deprecated, use reindex
```
from matplotlib.colors import ListedColormap

df.set_index('App')\
  .reindex(df.set_index('App').sum().sort_values().index, axis=1)\
  .T.plot(kind='bar', stacked=True,
          colormap=ListedColormap(sns.color_palette("GnBu", 10)),
          figsize=(12,6))
```
Output:


Question:
How to convert JSON data inside a pandas column into new columns I have this short version of ADSB json data and would like to convert it into dataFrame columns as Icao, Alt, Lat, Long, Spd, Cou..... After Alperen told me to do this
```
df = pd.read_json('2016-06-20-2359Z.json', lines=True),
```
I can load it into a DataFrame. However, df.acList is
```
[{'Id': 10537990, 'Rcvr': 1, 'HasSig': False, ...
Name: acList, dtype: object
```
How can I get the Icao, Alt, Lat, Long, Spd, Cou data?
```json
"src":1,
   "feeds":[
      {
         "id":1,
         "name":"ADSBexchange.com",
         "polarPlot":false
      }
   ],
   "srcFeed":1,
   "showSil":true,
   "showFlg":true,
   "showPic":true,
   "flgH":20,
   "flgW":85,
   "acList":[
      {
         "Id":11281748,
         "Rcvr":1,
         "HasSig":false,
         "Icao":"AC2554",
         "Bad":false,
         "Reg":"N882AS",
         "FSeen":"\/Date(1466467166951)\/",
         "TSecs":3,
         "CMsgs":1,
         "AltT":0,
         "Tisb":false,
         "TrkH":false,
         "Type":"CRJ2",
         "Mdl":"2001
BOMBARDIER INC
 CL-600-2B19",
         "Man":"Bombardier",
         "CNum":"7503",
         "Op":"EXPRESSJET AIRLINES INC - ATLANTA, GA",
         "OpIcao":"ASQ",
         "Sqk":"",
         "VsiT":0,
         "WTC":2,
         "Species":1,
         "Engines":"2",
         "EngType":3,
         "EngMount":1,
         "Mil":false,
         "Cou":"United States",
         "HasPic":false,
         "Interested":false,
         "FlightsCount":0,
         "Gnd":false,
         "SpdTyp":0,
         "CallSus":false,
         "TT":"a",
         "Trt":1,
         "Year":"2001"
      },
      {
         "Id":11402205,
         "Rcvr":1,
         "HasSig":true,
         "Sig":110,
         "Icao":"ADFBDD",
         "Bad":false,
         "FSeen":"\/Date(1466391940977)\/",
         "TSecs":75229,
         "CMsgs":35445,
         "Alt":8025,
         "GAlt":8025,
         "AltT":0,
         "Call":"TEST1234",
         "Tisb":false,
         "TrkH":false,
         "Sqk":"0262",
         "Help":false,
         "VsiT":0,
         "WTC":0,
         "Species":0,
         "EngType":0,
         "EngMount":0,
         "Mil":true,
         "Cou":"United States",
         "HasPic":false,
         "Interested":false,
         "FlightsCount":0,
         "Gnd":true,
         "SpdTyp":0,
         "CallSus":false,
         "TT":"a",
         "Trt":1
      }
   ],
   "totalAc":4231,
   "lastDv":"636019887431643594",
   "shtTrlSec":61,
   "stm":1466467170029
}
</pre>
```

Response:
If you already have your data in acList column in a pandas DataFrame, simply do:
```
import pandas as pd
pd.io.json.json_normalize(df.acList[0])

Alt AltT    Bad CMsgs   CNum    Call    CallSus Cou EngMount    EngType ... Sqk TSecs   TT  Tisb    TrkH    Trt Type    VsiT    WTC Year
0   NaN 0   False   1   7503    NaN False   United States   1   3   ...     3   a   False   False   1   CRJ2    0   2   2001
1   8025.0  0   False   35445   NaN TEST1234    False   United States   0   0   ... 0262    75229   a   False   False   1   NaN 0   0   NaN
```
Since pandas 1.0 the imports should be:
```
import pandas as pd
pd.json_normalize(df.acList[0])
```


Question: SKLearn MinMaxScaler - scale specific columns only [duplicate] This question already has answers here: pandas dataframe columns scaling with sklearn (9 answers) Closed 3 years ago. I'd like to scale some (but not all) of the columns in a Pandas dataFrame using a MinMaxScaler. How can I do it?

Response:
Demo:
```

```python
df = pd.DataFrame(np.random.randn(5, 3), index=list('abcde'), columns=list('xyz'))
```

```python
df
#Output
#x         y         z
#a -0.325882 -0.299432 -0.182373
#b -0.833546 -0.472082  1.158938
#c -0.328513 -0.664035  0.789414
#d -0.031630 -1.040802 -1.553518
#e  0.813328  0.076450  0.022122
```

```python
from sklearn.preprocessing import MinMaxScaler
```

```python
mms = MinMaxScaler()
```

```python
df[['x','z']] = mms.fit_transform(df[['x','z']])
```

```python
df
#Output
#x         y         z
#a  0.308259 -0.299432  0.505500
#b  0.000000 -0.472082  1.000000
#c  0.306662 -0.664035  0.863768
#d  0.486932 -1.040802  0.000000
#e  1.000000  0.076450  0.580891
#```
#the same result can be also achieved using sklearn.preprocessing.minmax_scale:
#```
#from sklearn.preprocessing import minmax_scale
#
#df[['x','z']] = minmax_scale(df[['x','z']])
#```
```


Question:
Change timezone of date-time column in pandas and add as hierarchical index I have data with a time-stamp in UTC. I'd like to convert the timezone of this timestamp to 'US/Pacific' and add it as a hierarchical index to a pandas DataFrame. I've been able to convert the timestamp as an Index, but it loses the timezone formatting when I try to add it back into the DataFrame, either as a column or as an index.
```

```python
import pandas as pd
```

```python
dat = pd.DataFrame({'label':['a', 'a', 'a', 'b', 'b', 'b'], 'datetime':['2011-07-19 07:00:00', '2011-07-19 08:00:00', '2011-07-19 09:00:00', '2011-07-19 07:00:00', '2011-07-19 08:00:00', '2011-07-19 09:00:00'], 'value':range(6)})
```

```python
dat.dtypes
#Output
##datetime    object
##label       object
##value        int64
##dtype: object
#```
#Now if I try to convert the Series directly I run into an error.
#```
```

```python
times = pd.to_datetime(dat['datetime'])
```

```python
times.tz_localize('UTC')
#Output
##Traceback (most recent call last):
##  File "<stdin>", line 1, in <module>
##  File "/Users/erikshilts/workspace/schedule-detection/python/pysched/env/lib/python2.7/site-packages/pandas/core/series.py", line 3170, in tz_localize
##    raise Exception('Cannot tz-localize non-time series')
##Exception: Cannot tz-localize non-time series
#```
#If I convert it to an Index then I can manipulate it as a timeseries. Notice that the index now has the Pacific timezone.
#```
```

```python
times_index = pd.Index(times)
```

```python
times_index_pacific = times_index.tz_localize('UTC').tz_convert('US/Pacific')
```

```python
times_index_pacific
#Output
##<class 'pandas.tseries.index.DatetimeIndex'>
##[2011-07-19 00:00:00, ..., 2011-07-19 02:00:00]
##Length: 6, Freq: None, Timezone: US/Pacific
#```
#However, now I run into problems adding the index back to the dataframe as it loses its timezone formatting:
#```
```

```python
dat_index = dat.set_index([dat['label'], times_index_pacific])
```

```python
dat_index
#Output
##                                      datetime label  value
##label
##a     2011-07-19 07:00:00  2011-07-19 07:00:00     a      0
##      2011-07-19 08:00:00  2011-07-19 08:00:00     a      1
##      2011-07-19 09:00:00  2011-07-19 09:00:00     a      2
##b     2011-07-19 07:00:00  2011-07-19 07:00:00     b      3
##      2011-07-19 08:00:00  2011-07-19 08:00:00     b      4
##      2011-07-19 09:00:00  2011-07-19 09:00:00     b      5
#```
#You'll notice the index is back on the UTC timezone instead of the converted Pacific timezone. How can I change the timezone and add it as an index to a DataFrame?
```

Response:
If you set it as the index, it's automatically converted to an Index:
```

```python
dat.index = pd.to_datetime(dat.pop('datetime'), utc=True)
```

```python
dat
#Output
#label  value
#datetime
#2011-07-19 07:00:00     a      0
#2011-07-19 08:00:00     a      1
#2011-07-19 09:00:00     a      2
#2011-07-19 07:00:00     b      3
#2011-07-19 08:00:00     b      4
#2011-07-19 09:00:00     b      5
#```
#Then do the tz_localize:
#```
```

```python
dat.index = dat.index.tz_localize('UTC').tz_convert('US/Pacific')
```

```python
dat
#Output
#label  value
#datetime
#2011-07-19 00:00:00-07:00     a      0
#2011-07-19 01:00:00-07:00     a      1
#2011-07-19 02:00:00-07:00     a      2
#2011-07-19 00:00:00-07:00     b      3
#2011-07-19 01:00:00-07:00     b      4
#2011-07-19 02:00:00-07:00     b      5
#```
#And then you can append the label column to the index: Hmmm this is definitely a bug!
#```
```

```python
dat.set_index('label', append=True).swaplevel(0, 1)
#Output
#value
#label datetime
#a     2011-07-19 07:00:00      0
#      2011-07-19 08:00:00      1
#      2011-07-19 09:00:00      2
#b     2011-07-19 07:00:00      3
#      2011-07-19 08:00:00      4
#      2011-07-19 09:00:00      5
#```
#A hacky workaround is to convert the (datetime) level directly (when it's already a MultiIndex):
#```
```

```python
dat.index.levels[1] = dat.index.get_level_values(1).tz_localize('UTC').tz_convert('US/Pacific')
```

```python
dat1
#Output
#value
#label datetime
#a     2011-07-19 00:00:00-07:00      0
#      2011-07-19 01:00:00-07:00      1
#      2011-07-19 02:00:00-07:00      2
#b     2011-07-19 00:00:00-07:00      3
#      2011-07-19 01:00:00-07:00      4
#      2011-07-19 02:00:00-07:00      5
#```
```


Question:
Pandas filter data frame rows by function I want to filter a dataframe by a more complex function based on different values in the row. Is there a possibility to filter DF rows by a boolean function like you can do it e.g. in ES6 filter function? Extreme simplified example to illustrate the problem:
```py
import pandas as pd

def filter_fn(row):
    if row['Name'] == 'Alisa' and row['Age'] > 24:
        return False

    return row

d = {
    'Name': ['Alisa', 'Bobby', 'jodha', 'jack', 'raghu', 'Cathrine',
             'Alisa', 'Bobby', 'kumar', 'Alisa', 'Alex', 'Cathrine'],
    'Age': [26, 24, 23, 22, 23, 24, 26, 24, 22, 23, 24, 24],

    'Score': [85, 63, 55, 74, 31, 77, 85, 63, 42, 62, 89, 77]}

df = pd.DataFrame(d, columns=['Name', 'Age', 'Score'])

df = df.apply(filter_fn, axis=1, broadcast=True)
```
I found something using apply() but this actually returns only False/True filled rows using a bool function, which is expected. My workaround would be returning the row itself when the function result would be True and returning False if not. But this would require an additional filtering after that.
```none
Name    Age  Score
0      False  False  False
1      Bobby     24     63
2      jodha     23     55
3       jack     22     74
4      raghu     23     31
5   Cathrine     24     77
6      False  False  False
7      Bobby     24     63
8      kumar     22     42
9      Alisa     23     62
10      Alex     24     89
11  Cathrine     24     77
```

Response:
I think using functions here is unnecessary. It is better and mainly faster to use boolean indexing:
```
m = (df['Name'] == 'Alisa') & (df['Age'] > 24)
print(m)
0      True
1     False
2     False
3     False
4     False
5     False
6      True
7     False
8     False
9     False
10    False
11    False
dtype: bool

#invert mask by ~
df1 = df[~m]
```
For more complicated filtering, you could use a function which must return a boolean value:
```
def filter_fn(row):
    if row['Name'] == 'Alisa' and row['Age'] > 24:
        return False
    else:
        return True

df = pd.DataFrame(d, columns=['Name', 'Age', 'Score'])
m = df.apply(filter_fn, axis=1)
print(m)
0     False
1      True
2      True
3      True
4      True
5      True
6     False
7      True
8      True
9      True
10     True
11     True
dtype: bool

df1 = df[m]
```


Question:
Pandas' merge returns a column with _x appended to the name I have to dataframes, df1 has columns A, B, C, D... and df2 has columns A, B, E, F... The keys I want to merge with are in column A. B is also (most likely) the same in both dataframes. This is a big dataset so I do not have a good overview of everything yet. When I do
```py
pd.merge(df1, df2, on='A')
```
the results contains a column called B_x. Since the dataset is big and messy I haven't tried to investigate how B_x differs from B in df1 and B in df2. So my question is just in general: What does Pandas mean when it has appended _x to a column name in the merged dataframe?

Response:
The suffixes are added for any clashes in column names that are not involved in the merge operation, see online docs. So in your case if you think that they are same you could just do the merge on both columns:
```
pd.merge(df1, df2, on=['A', 'B'])
```
What this will do though is return only the values where A and B exist in both dataframes as the default merge type is an inner merge. So what you could do is compare this merged df size with your first one and see if they are the same and if so you could do a merge on both columns or just drop/rename the _x/_y suffix B columns. I would spend time though determining if these values are indeed the same and exist in both dataframes, in which case you may wish to perform an outer merge:
```
pd.merge(df1, df2, on=['A', 'B'], how='outer')
```
Then what you could do is then drop duplicate rows (and possibly any NaN rows) and that should give you a clean merged dataframe.
```
merged_df.drop_duplicates(cols=['A', 'B'],inplace=True)
```
See online docs for drop_duplicates


Question: "Anti-merge" in pandas (Python) How can I pick out the difference between to columns of the same name in two dataframes? I mean I have dataframe A with a column named X and dataframe B with column named X, if i do pd.merge(A, B, on=['X']), i'll get the common X values of A and B, but how can i get the "non-common" ones?

Response:
If you change the merge type to how='outer' and indicator=True this will add a column to tell you whether the values are left/both/right only:
```

```python
A = pd.DataFrame({'x':np.arange(5)})
B = pd.DataFrame({'x':np.arange(3,8)})
print(A)
print(B)
   x
0  0
1  1
2  2
3  3
4  4
   x
0  3
1  4
2  5
3  6
4  7
```

```python
pd.merge(A,B, how='outer', indicator=True)
#Output
#x      _merge
#0  0.0   left_only
#1  1.0   left_only
#2  2.0   left_only
#3  3.0        both
#4  4.0        both
#5  5.0  right_only
#6  6.0  right_only
#7  7.0  right_only
#```
#You can then filter the resultant merged df on the _merge col:
#```
```

```python
merged = pd.merge(A,B, how='outer', indicator=True)
merged[merged['_merge'] == 'left_only']
#Output
#x     _merge
#0  0.0  left_only
#1  1.0  left_only
#2  2.0  left_only
#```
#You can also use isin and negate the mask to find values not in B:
#```
```

```python
A[~A['x'].isin(B['x'])]
#Output
#x
#0  0
#1  1
#2  2
#```
```


Question:
Dropping duplicates in Pandas excluding one column This seems simple, but I can not find any information on it on the internet. I have a dataframe like below:
```
City    State Zip           Date        Description
Earlham IA    50072-1036    2014-10-10  Postmarket Assurance: Devices
Earlham IA    50072-1036    2014-10-10  Compliance: Devices
Madrid  IA    50156-1748    2014-09-10  Drug Quality Assurance
```
How can I eliminate duplicates that match 4 of 5 columns? The column not matching being Description. The result would be
```
City    State Zip           Date        Description
Earlham IA    50072-1036    2014-10-10  Postmarket Assurance: Devices
Madrid  IA    50156-1748    2014-09-10  Drug Quality Assurance
```
I found online that drop_duplicates with the subset parameter could work, but I am unsure of how I can apply it to multiple columns.

Response:
You've actually found the solution. For multiple columns, subset will be a list.
```
df.drop_duplicates(subset=['City', 'State', 'Zip', 'Date'])
```
Or, just by stating the column to be ignored:
```
df.drop_duplicates(subset=df.columns.difference(['Description']))
```


Question:
Python pandas integer YYYYMMDD to datetime I have a DataFrame that looks like the following:
```none
OrdNo  LstInvDt
9      20070620
11     20070830
19     20070719
21     20070719
23     20070719
26     20070911
29     20070918
31      0070816
34     20070925
```
LstInvDt of dtype int64. As you can see, the integers are of the format YYYYMMDD, e.g. 20070530 - 30th of May 2007. I have tried a range of approaches, the most obvious being; pd.to_datetime(dt['Date']) and pd.to_datetime(str(dt['Date'])) with multiple variations on the functions different parameters. The result has been that the date interpreted as being the time. The date is set to 1970-01-01 - outcome as per above example 1970-01-01 00:00:00.020070530 I also tried various .map() functions found in similar posts. How do I convert it correctly?

Response:
to_datetime accepts a format string:
```

```python
t = 20070530
pd.to_datetime(str(t), format='%Y%m%d')
#Output
#Timestamp('2007-05-30 00:00:00')
#```
#example:
#```
```

```python
t = 20070530
df = pd.DataFrame({'date':[t]*10})
df
#Output
#date
#0  20070530
#1  20070530
#2  20070530
#3  20070530
#4  20070530
#5  20070530
#6  20070530
#7  20070530
#8  20070530
#9  20070530
```

```python
df['DateTime'] = df['date'].apply(lambda x: pd.to_datetime(str(x), format='%Y%m%d'))
df
#Output
#date   DateTime
#0  20070530 2007-05-30
#1  20070530 2007-05-30
#2  20070530 2007-05-30
#3  20070530 2007-05-30
#4  20070530 2007-05-30
#5  20070530 2007-05-30
#6  20070530 2007-05-30
#7  20070530 2007-05-30
#8  20070530 2007-05-30
#9  20070530 2007-05-30
```

```python
df.dtypes
#Output
#date                 int64
#DateTime    datetime64[ns]
#dtype: object
#```
#EDIT Actually it's quicker to convert the type to string and then convert the entire series to a datetime rather than calling apply on every value:
#```
```

```python
df['DateTime'] = pd.to_datetime(df['date'].astype(str), format='%Y%m%d')
df
#Output
#date   DateTime
#0  20070530 2007-05-30
#1  20070530 2007-05-30
#2  20070530 2007-05-30
#3  20070530 2007-05-30
#4  20070530 2007-05-30
#5  20070530 2007-05-30
#6  20070530 2007-05-30
#7  20070530 2007-05-30
#8  20070530 2007-05-30
#9  20070530 2007-05-30
#```
#timings
#```
```

```python
%timeit df['date'].apply(lambda x: pd.to_datetime(str(x), format='%Y%m%d'))

100 loops, best of 3: 2.55 ms per loop
```

```python
%timeit pd.to_datetime(df['date'].astype(str), format='%Y%m%d')
1000 loops, best of 3: 396 µs per loop
```
```


Question:
Python/Pandas convert string to time only I have the following Pandas dataframe in Python 2.7.
```
import pandas as pd
trial_num = [1,2,3,4,5]
sail_rem_time = ['11:33:11','16:29:05','09:37:56','21:43:31','17:42:06']
dfc = pd.DataFrame(zip(*[trial_num,sail_rem_time]),columns=['Temp_Reading','Time_of_Sail'])
print dfc
```
The dataframe looks like this:
```
Temp_Reading Time_of_Sail
             1     11:33:11
             2     16:29:05
             3     09:37:56
             4     21:43:31
             5     17:42:06
```
This dataframe comes from a *.csv file. I use Pandas to read in the *.csv file as a Pandas dataframe. When I use print dfc.dtypes, it shows me that the column Time_of_Sail has a datatype object. I would like to convert this column to datetime datatype BUT I only want the time part - I don't want the year, month, date. I can try this:
```
dfc['Time_of_Sail'] = pd.to_datetime(dfc['Time_of_Sail'])
dfc['Time_of_Sail'] = [time.time() for time in dfc['Time_of_Sail']]
```
but the problem is that the when I run print dfc.dtypes it still shows that the column Time_of_Sail is object. Is there a way to convert this column into a datetime format that only has the time? Additional Information: To create the above dataframe and output, this also works:
```
import pandas as pd
trial_num = [1,2,3,4,5]
sail_rem_time = ['11:33:11','16:29:05','09:37:56','21:43:31','17:42:06']
data = [
    [trial_num[0],sail_rem_time[0]],
    [trial_num[1],sail_rem_time[1]],[trial_num[2],sail_rem_time[2]],
    [trial_num[3],sail_rem_time[3]]
    ]
dfc = pd.DataFrame(data,columns=['Temp_Reading','Time_of_Sail'])
dfc['Time_of_Sail'] = pd.to_datetime(dfc['Time_of_Sail'])
dfc['Time_of_Sail'] = [time.time() for time in dfc['Time_of_Sail']]
print dfc
print dfc.dtypes
```

Response:
These two lines:
```
dfc['Time_of_Sail'] = pd.to_datetime(dfc['Time_of_Sail'])
dfc['Time_of_Sail'] = [time.time() for time in dfc['Time_of_Sail']]
```
Can be written as:
```
dfc['Time_of_Sail'] = pd.to_datetime(dfc['Time_of_Sail'],format= '%H:%M:%S' ).dt.time
```


Question: Drop a specific row in Pandas I tried drop method of pandas but I didn't use it. How do I remove a specific row in pandas with Python? e.g.: My specific row is => Name: Bertug Grade: A Age: 15

Response:
```
df = pd.DataFrame([['Jhon',15,'A'],['Anna',19,'B'],['Paul',25,'D']])
df. columns = ['Name','Age','Grade']

df
Out[472]:
   Name  Age Grade
0  Jhon   15     A
1  Anna   19     B
2  Paul   25     D
```
You can get the index of your row:
```
i = df[((df.Name == 'jhon') &( df.Age == 15) & (df.Grade == 'A'))].index
```
and then drop it:
```
df.drop(i)
Out[474]:
   Name  Age Grade
1  Anna   19     B
2  Paul   25     D
```
As @jezrael pointed our, you can also just negate all three:
```
df[((df.Name != 'jhon') &( df.Age != 15) & (df.Grade != 'A'))]
Out[477]:
   Name  Age Grade
1  Anna   19     B
2  Paul   25     D
```


Question:
Reading a csv with a timestamp column, with pandas When doing:
```
import pandas
x = pandas.read_csv('data.csv', parse_dates=True, index_col='DateTime',
                                names=['DateTime', 'X'], header=None, sep=';')
```
with this data.csv file:
```
1449054136.83;15.31
1449054137.43;16.19
1449054138.04;19.22
1449054138.65;15.12
1449054139.25;13.12
```
(the 1st colum is a UNIX timestamp, i.e. seconds elapsed since 1/1/1970), I get this error when resampling the data every 15 second with x.resample('15S'):
```
TypeError: Only valid with DatetimeIndex, TimedeltaIndex or PeriodIndex
```
It's like the "datetime" information has not been parsed:
```
X
DateTime
1.449054e+09  15.31
1.449054e+09  16.19
...
```
How to import a .CSV with date stored as timestamp with pandas module? Then once I will be able to import the CSV, how to access to the lines for which date > 2015-12-02 12:02:18 ?

Response:
Use to_datetime and pass unit='s' to parse the units as unix timestamps, this will be much faster:
```

```python
pd.to_datetime(df.index, unit='s')
#Output
#DatetimeIndex(['2015-12-02 11:02:16.830000', '2015-12-02 11:02:17.430000',
#               '2015-12-02 11:02:18.040000', '2015-12-02 11:02:18.650000',
#               '2015-12-02 11:02:19.250000'],
#              dtype='datetime64[ns]', name=0, freq=None)
#```
#Timings:
#```
```

```python
import time
%%timeit
import time
def date_parser(string_list):
    return [time.ctime(float(x)) for x in string_list]

df = pd.read_csv(io.StringIO(t), parse_dates=[0],  sep=';',
                 date_parser=date_parser,
                 index_col='DateTime',
                 names=['DateTime', 'X'], header=None)
100 loops, best of 3: 4.07 ms per loop
```
and
```
```

```python
%%timeit
t="""1449054136.83;15.31
1449054137.43;16.19
1449054138.04;19.22
1449054138.65;15.12
1449054139.25;13.12"""
df = pd.read_csv(io.StringIO(t), header=None, sep=';', index_col=[0])
df.index = pd.to_datetime(df.index, unit='s')
100 loops, best of 3: 1.69 ms per loop
```
So using to_datetime is over 2x faster on this small dataset, I expect this to scale much better than the other methods
```


Question:
How to drop columns which have same values in all rows via pandas or spark dataframe? Suppose I've data similar to following:
```
index id   name  value  value2  value3  data1  val5
    0  345  name1    1      99      23     3      66
    1   12  name2    1      99      23     2      66
    5    2  name6    1      99      23     7      66
```
How can we drop all those columns like (value, value2, value3) where all rows have the same values, in one command or couple of commands using python? Consider we have many columns similar to value, value2, value3...value200. Output:
```
index    id  name   data1
       0   345  name1    3
       1    12  name2    2
       5     2  name6    7
```

Response:
What we can do is use nunique to calculate the number of unique values in each column of the dataframe, and drop the columns which only have a single unique value:
```

```python
nunique = df.nunique()
cols_to_drop = nunique[nunique == 1].index
df.drop(cols_to_drop, axis=1)
#Output
#index   id   name  data1
#0      0  345  name1      3
#1      1   12  name2      2
#2      5    2  name6      7
#```
#Another way is to just diff the numeric columns, take abs values and sums them:
#```
```

```python
cols = df.select_dtypes([np.number]).columns
diff = df[cols].diff().abs().sum()
df.drop(diff[diff== 0].index, axis=1)

#Output
#index   id   name  data1
#0      0  345  name1      3
#1      1   12  name2      2
#2      5    2  name6      7
#```
#Another approach is to use the property that the standard deviation will be zero for a column with the same value:
#```
```

```python
cols = df.select_dtypes([np.number]).columns
std = df[cols].std()
cols_to_drop = std[std==0].index
df.drop(cols_to_drop, axis=1)
#Output
#index   id   name  data1
#0      0  345  name1      3
#1      1   12  name2      2
#2      5    2  name6      7
#```
#Actually the above can be done in a one-liner:
#```
```

```python
df.drop(df.std()[(df.std() == 0)].index, axis=1)
#Output
#index   id   name  data1
#0      0  345  name1      3
#1      1   12  name2      2
#2      5    2  name6      7
#```
```


Question:
Pandas compare next row I have a dataframe like this
```
d={}
d['z']=['Q8','Q8','Q7','Q9','Q9']
d['t']=['10:30','10:31','10:38','10:40','10:41']
d['qty']=[20,20,9,12,12]
```
I want compare first row with second row is qty same as next row AND is t greater in the next row AND is z value same as next row The desired value is
```
qty                   t   z  valid
0   20 2015-06-05 10:30:00  Q8  False
1   20 2015-06-05 10:31:00  Q8   True
2    9 2015-06-05 10:38:00  Q7  False
3   12 2015-06-05 10:40:00  Q9  False
4   12 2015-06-05 10:41:00  Q9   True
```

Response:
Looks like you want to use the Series.shift method. Using this method, you can generate new columns which are offset to the original columns. Like this:
```
df['qty_s'] = df['qty'].shift(-1)
df['t_s'] = df['t'].shift(-1)
df['z_s'] = df['z'].shift(-1)
```
Now you can compare these:
```
df['is_something'] = (df['qty'] == df['qty_s']) & (df['t'] < df['t_s']) & (df['z'] == df['z_s'])
```
Here is a simplified example of how Series.shift works to compare next row to the current:
```
df = pd.DataFrame({"temp_celcius":pd.np.random.choice(10, 10) + 20}, index=pd.date_range("2015-05-15", "2015-05-24"))
df
            temp_celcius

2015-05-15            21
2015-05-16            28
2015-05-17            27
2015-05-18            21
2015-05-19            25
2015-05-20            28
2015-05-21            25
2015-05-22            22
2015-05-23            29
2015-05-24            25

df["temp_c_yesterday"] = df["temp_celcius"].shift(1)
df
            temp_celcius  temp_c_yesterday
2015-05-15            21               NaN
2015-05-16            28                21
2015-05-17            27                28
2015-05-18            21                27
2015-05-19            25                21
2015-05-20            28                25
2015-05-21            25                28
2015-05-22            22                25
2015-05-23            29                22
2015-05-24            25                29

df["warmer_than_yesterday"] = df["temp_celcius"] > df["temp_c_yesterday"]
            temp_celcius  temp_c_yesterday warmer_than_yesterday
2015-05-15            21               NaN                 False
2015-05-16            28                21                  True
2015-05-17            27                28                 False
2015-05-18            21                27                 False
2015-05-19            25                21                  True
2015-05-20            28                25                  True
2015-05-21            25                28                 False
2015-05-22            22                25                 False
2015-05-23            29                22                  True
2015-05-24            25                29                 False
```
If I misunderstood your query, please post a comment and I'll update my answer.


Question:
How to access pandas DataFrame datetime index using strings This is a very simple and practical question. I have the feeling that it must be a silly detail and that there should be similar questions. I wasn't able to find them tho. If someone does I'll happily delete this one. The closest I found were these: pandas: iterating over DataFrame index with loc How to select rows within a pandas dataframe based on time only when index is date and time anyway, the thing is, I have a datetime indexed panda dataframe as follows:
```

```python
y
#Output
#PETR4  CSNA3  VALE5
#2008-01-01    0.0    0.0    0.0
#2008-01-02    1.0    1.0    1.0
#2008-01-03    7.0    7.0    7.0
```

```python
y.index
#Output
#DatetimeIndex(['2008-01-01', '2008-01-02', '2008-01-03'], dtype='datetime64[ns]', freq=None)
#```
#Oddly enough, I can't access its values using none of the following methods:
#```
```

```python
y[datetime.datetime(2008,1,1)]
```

```python
y['2008-1-1']
```

```python
y['1/1/2008']
```
I get the KeyError error. Even more weird is that the following methods DO work:
```
```

```python
y['2008']
#Output
#PETR4  CSNA3  VALE5
#2008-01-01    0.0    0.0    0.0
#2008-01-02    1.0    1.0    1.0
#2008-01-03    7.0    7.0    7.0
```

```python
y['2008-1']
#Output
#PETR4  CSNA3  VALE5
#2008-01-01    0.0    0.0    0.0
#2008-01-02    1.0    1.0    1.0
#2008-01-03    7.0    7.0    7.0
#```
#I'm fairly new to pandas so maybe I'm missing something here?
```

Response:
pandas is taking what's inside the [] and deciding what it should do. If it's a subset of column names, it'll return a DataFrame with those columns. If it's a range of index values, it'll return a subset of those rows. What is does not handle is taking a single index value. Solution Two work around's 1.Turn the argument into something pandas interprets as a range.
```
df['2008-01-01':'2008-01-01']
```
2.Use the method designed to give you this result. loc[]
```
df.loc['2008-01-01']
```
Link to the documentation


Question:
How to rotate X-axis labels in bokeh figure? I'm just starting to use Bokeh. Below I create some args I use for the rect figure.
```
x_length = var_results.index * 5.5
```
Multiplying the index by 5.5 gave me more room between labels.
```
names = var_results.Feature.tolist()
y_length = var_results.Variance
y_center = var_results.Variance/2
```
var_results is a Pandas dataframe that has a typical, sequential, non-repeating index. var_results also has a column Features that is strings of non-repeated, names, and finally it has a column Variance which is dtype float.
```
r = figure(x_range = names,
           y_range = (-0.05,.3),
           active_scroll = 'wheel_zoom',
           x_axis_label = 'Features',
           y_axis_label = 'Variance')

r.rect(x_length,
       y_center,
       width=1,
       height=y_length,
       color = "#ff1200")
output_notebook()
show(r)
```
I'm essentially making a bar chart with rectangles. Bokeh seems to be very customizable. But my graph looks rough around the edges, literally. As you can see there is an ugly smudge just below the chart and above the x-axis title 'Features'. This is the label titles (technically the rectangle titles). How do I create space for and perhaps rotate to 45 degrees the labels so that they are readable and not just an overlapping mess?

Response:
In order to rotate the labels e.g. by 90 degrees to the left, you can set major_label_orientation to π/2. This can be done either when creating the axis element (as a kwarg to the axis constructor if you are using low level plotting) or also after you have created a plot/figure, for instance by:
```
p.xaxis.major_label_orientation = math.pi/2

# or alternatively:
p.xaxis.major_label_orientation = "vertical"
```
See also this example in the documentation.