Question:
Pandas to_html() truncates string contents I have a Python Pandas DataFrame object containing textual data. My problem is, that when I use to_html() function, it truncates the strings in the output. For example:
```py
import pandas
df = pandas.DataFrame({'text': ['Lorem ipsum dolor sit amet, consectetur adipiscing elit.']})
print (df.to_html())
```
The output is truncated at adapis...
```
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>text</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td> Lorem ipsum dolor sit amet, consectetur adipis...</td>
    </tr>
  </tbody>
</table>
```
There is a related question on SO, but it uses placeholders and search/replace functionality to postprocess the HTML, which I would like to avoid: Writing full contents of Pandas dataframe to HTML table Is there a simpler solution to this problem? I could not find anything related from the documentation.

Response:
What you are seeing is pandas truncating the output for display purposes only. The default max_colwidth value is 50 which is what you are seeing. You can set this value to whatever you desire or you can set it to -1 which effectively turns this off:
```
pd.set_option('display.max_colwidth', -1)
```
Although I would advise against this, it would be better to set it to something that can be displayed easily in your console or ipython. A list of the options can be found here: http://pandas.pydata.org/pandas-docs/stable/options.html


Question:
Pandas apply but only for rows where a condition is met I would like to use Pandas df.apply but only for certain rows As an example, I want to do something like this, but my actual issue is a little more complicated:
```
import pandas as pd
import math
z = pd.DataFrame({'a':[4.0,5.0,6.0,7.0,8.0],'b':[6.0,0,5.0,0,1.0]})
z.where(z['b'] != 0, z['a'] / z['b'].apply(lambda l: math.log(l)), 0)
```
What I want in this example is the value in 'a' divided by the log of the value in 'b' for each row, and for rows where 'b' is 0, I simply want to return 0.

Response:
The other answers are excellent, but I thought I'd add one other approach that can be faster in some circumstances – using broadcasting and masking to achieve the same result:
```
import numpy as np

mask = (z['b'] != 0)
z_valid = z[mask]

z['c'] = 0
z.loc[mask, 'c'] = z_valid['a'] / np.log(z_valid['b'])
```
Especially with very large dataframes, this approach will generally be faster than solutions based on apply().


Question:
Pandas: Check if row exists with certain values I have a two dimensional (or more) pandas DataFrame like this:
```

```python
import pandas as pd
```

```python
df = pd.DataFrame([[0,1],[2,3],[4,5]], columns=['A', 'B'])
```

```python
df
#Output
#   A  B
#0  0  1
#1  2  3
#2  4  5
#```
#Now suppose I have a numpy array like np.array([2,3]) and want to check if there is any row in df that matches with the contents of my array. Here the answer should obviously true but eg. np.array([1,2]) should return false as there is no row with both 1 in column A and 2 in column B. Sure this is easy but don't see it right now.
```

Response:
Turns out it is really easy, the following does the job here:
```

```python
((df['A'] == 2) & (df['B'] == 3)).any()
#Output
#True
```

```python
((df['A'] == 1) & (df['B'] == 2)).any()
#Output
#False
#```
#Maybe somebody comes up with a better solution which allows directly passing in the array and the list of columns to match. Note that the parenthesis around df['A'] == 2 are not optional since the & operator binds just as strong as the == operator.
```


Question:
Logarithmic returns in pandas dataframe Python pandas has a pct_change function which I use to calculate the returns for stock prices in a dataframe:
```
ndf['Return']= ndf['TypicalPrice'].pct_change()
```
I am using the following code to get logarithmic returns, but it gives the exact same values as the pct.change() function:
```
ndf['retlog']=np.log(ndf['TypicalPrice'].astype('float64')/ndf['TypicalPrice'].astype('float64').shift(1))
#np is for numpy
```

Response:
Here is one way to calculate log return using .shift(). And the result is similar to but not the same as the gross return calculated by pct_change(). Can you upload a copy of your sample data (dropbox share link) to reproduce the inconsistency you saw?
```
import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame(100 + np.random.randn(100).cumsum(), columns=['price'])
df['pct_change'] = df.price.pct_change()
df['log_ret'] = np.log(df.price) - np.log(df.price.shift(1))

Out[56]:
       price  pct_change  log_ret
0   101.7641         NaN      NaN
1   102.1642      0.0039   0.0039
2   103.1429      0.0096   0.0095
3   105.3838      0.0217   0.0215
4   107.2514      0.0177   0.0176
5   106.2741     -0.0091  -0.0092
6   107.2242      0.0089   0.0089
7   107.0729     -0.0014  -0.0014
..       ...         ...      ...
92  101.6160      0.0021   0.0021
93  102.5926      0.0096   0.0096
94  102.9490      0.0035   0.0035
95  103.6555      0.0069   0.0068
96  103.6660      0.0001   0.0001
97  105.4519      0.0172   0.0171
98  105.5788      0.0012   0.0012
99  105.9808      0.0038   0.0038

[100 rows x 3 columns]
```


Question:
TypeError: bad operand type for unary ~: float [duplicate] This question already has answers here: Ignoring NaNs with str.contains (6 answers) Closed last year.
```
df = df[~df["column"].str.contains("Total")]

TypeError: bad operand type for unary ~: 'float'
```
Why does .str.contains() return a float? What should I be doing here?

Response:
I think there are NaNs values, so need specify parameter na:
```
df = pd.DataFrame({
    'column': ['Total','a',np.nan],
    'B': list(range(3))
})
print (df)
  column  B
0  Total  0
1      a  1
2    NaN  2

df = df[~df["column"].str.contains("Total", na=False)]
print (df)
  column  B
1      a  1
2    NaN  2
```


Question:
python pandas flatten a dataframe to a list I have a df like so:
```
import pandas
a=[['1/2/2014', 'a', '6', 'z1'],
   ['1/2/2014', 'a', '3', 'z1'],
   ['1/3/2014', 'c', '1', 'x3'],
   ]
df = pandas.DataFrame.from_records(a[1:],columns=a[0])
```
I want to flatten the df so it is one continuous list like so: ['1/2/2014', 'a', '6', 'z1', '1/2/2014', 'a', '3', 'z1','1/3/2014', 'c', '1', 'x3'] I can loop through the rows and extend to a list, but is a much easier way to do it?

Response:
You can use .flatten() on the DataFrame converted to a NumPy array:
```
df.to_numpy().flatten()
```
and you can also add .tolist() if you want the result to be a Python list. Edit In previous versions of Pandas, the values attributed was used instead of the .to_numpy() method, as mentioned in the comments below.


Question:
How to update values in a specific row in a Python Pandas DataFrame? With the nice indexing methods in Pandas I have no problems extracting data in various ways. On the other hand I am still confused about how to change data in an existing DataFrame. In the following code I have two DataFrames and my goal is to update values in a specific row in the first df from values of the second df. How can I achieve this?
```
import pandas as pd
df = pd.DataFrame({'filename' :  ['test0.dat', 'test2.dat'],
                                  'm': [12, 13], 'n' : [None, None]})
df2 = pd.DataFrame({'filename' :  'test2.dat', 'n':16}, index=[0])

# this overwrites the first row but we want to update the second
# df.update(df2)

# this does not update anything
df.loc[df.filename == 'test2.dat'].update(df2)

print(df)
```
gives
```
filename   m     n
0  test0.dat  12  None
1  test2.dat  13  None

[2 rows x 3 columns]
```
but how can I achieve this:
```
filename   m     n
0  test0.dat  12  None
1  test2.dat  13  16

[2 rows x 3 columns]
```

Response:
So first of all, pandas updates using the index. When an update command does not update anything, check both left-hand side and right-hand side. If you don't update the indices to follow your identification logic, you can do something along the lines of
```

```python
df.loc[df.filename == 'test2.dat', 'n'] = df2[df2.filename == 'test2.dat'].loc[0]['n']
```

```python
df
#Output
#Out[331]:
#    filename   m     n
#0  test0.dat  12  None
#1  test2.dat  13    16
#```
#If you want to do this for the whole table, I suggest a method I believe is superior to the previously mentioned ones: since your identifier is filename, set filename as your index, and then use update() as you wanted to. Both merge and the apply() approach contain unnecessary overhead:
#```
```

```python
df.set_index('filename', inplace=True)
```

```python
df2.set_index('filename', inplace=True)
```

```python
df.update(df2)
```

```python
df
#Output
#Out[292]:
#            m     n
#filename
#test0.dat  12  None
#test2.dat  13    16
#```
```


Question:
Python Pandas Group by date using datetime data I have a datetime column Date_Time that I wish to groupby without creating a new column. Is this possible? I tried the following and it does not work.
```
df = pd.groupby(df,by=[df['Date_Time'].date()])
```

Response:
You can use groupby by dates of column Date_Time by dt.date:
```
df = df.groupby([df['Date_Time'].dt.date]).mean()
```
Sample:
```
df = pd.DataFrame({'Date_Time': pd.date_range('10/1/2001 10:00:00', periods=3, freq='10H'),
                   'B':[4,5,6]})

print (df)
   B           Date_Time
0  4 2001-10-01 10:00:00
1  5 2001-10-01 20:00:00
2  6 2001-10-02 06:00:00

print (df['Date_Time'].dt.date)
0    2001-10-01
1    2001-10-01
2    2001-10-02
Name: Date_Time, dtype: object

df = df.groupby([df['Date_Time'].dt.date])['B'].mean()
print(df)
Date_Time
2001-10-01    4.5
2001-10-02    6.0
Name: B, dtype: float64
```
Another solution with resample:
```
df = df.set_index('Date_Time').resample('D')['B'].mean()

print(df)
Date_Time
2001-10-01    4.5
2001-10-02    6.0
Freq: D, Name: B, dtype: float64
```


Question: Order categories by count in a countplot I know that seaborn.countplot has the attribute order which can be set to determine the order of the categories. But what I would like to do is have the categories be in order of descending count. I know that I can accomplish this by computing the count manually (using a groupby operation on the original dataframe, etc.) but I am wondering if this functionality exists with seaborn.countplot.

Response:
This functionality is not built into seaborn.countplot as far as I know - the order parameter only accepts a list of strings for the categories, and leaves the ordering logic to the user. This is not hard to do with value_counts() provided you have a DataFrame though. For example,
```
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

sns.set(style='darkgrid')

titanic = sns.load_dataset('titanic')
sns.countplot(x = 'class',
              data = titanic,
              order = titanic['class'].value_counts().index)
plt.show()
```


Question:
AttributeError: Can only use .str accessor with string values, which use np.object_ dtype in pandas Str.replace method returns an attribute error.
```
dc_listings['price'].str.replace(',', '')
AttributeError: Can only use .str accessor with string values, which use np.object_ dtype in pandas
```
Here are the top 5 rows of my price column.
```

```python
dc_listings['price'].head()
#Output
#577     185.0
#2166    180.0
#3631    175.0
#71      128.0
#1011    115.0
#Name: price, dtype: float64
#```
#This stack overflow thread recommends to check if my column has NAN values but non of the values in my column are NAN.
#```
```

```python
dc_listings[dc_listings['price'].isnull()]
#Output
#host_response_rate host_acceptance_rate ... price cleaning_fee security_deposit
#```
```

Response:
As the error states, you can only use .str with string columns, and you have a float64. There won't be any commas in a float, so what you have won't really do anything, but in general, you could cast it first:
```
dc_listings['price'].astype(str).str.replace...
```
For example:
```

```python
df
#Output
#a         b         c         d         e
#0  0.645821  0.152197  0.006956  0.600317  0.239679
#1  0.865723  0.176842  0.226092  0.416990  0.290406
#2  0.046243  0.931584  0.020109  0.374653  0.631048
#3  0.544111  0.967388  0.526613  0.794931  0.066736
#4  0.528742  0.670885  0.998077  0.293623  0.351879
```

```python
df['a'].astype(str).str.replace("5", " hi ")
#Output
#0    0.64 hi 8208 hi  hi 4779467
#1          0.86 hi 7231174332336
#2            0.04624337481411367
#3       0. hi 44111244991 hi 194
#4          0. hi 287421814241892
#Name: a, dtype: object
#```
```


Question:
How to remove nan value while combining two column in Panda Data frame? I am trying but not able to remove nan while combining two columns of a DataFrame. Data is like:
```
feedback_id                  _id
568a8c25cac4991645c287ac     nan
568df45b177e30c6487d3603     nan
nan                          568df434832b090048f34974
nan                          568cd22e9e82dfc166d7dff1
568df3f0832b090048f34711     nan
nan                          568e5a38b4a797c664143dda
```
I want:
```
feedback_request_id
568a8c25cac4991645c287ac
568df45b177e30c6487d3603
568df434832b090048f34974
568cd22e9e82dfc166d7dff1
568df3f0832b090048f34711
568e5a38b4a797c664143dda
```
Here is my code:
```
df3['feedback_request_id'] = ('' if df3['_id'].empty else df3['_id'].map(str)) + ('' if df3['feedback_id'].empty else df3['feedback_id'].map(str))
```
Output I'm getting:
```
feedback_request_id
568a8c25cac4991645c287acnan
568df45b177e30c6487d3603nan
nan568df434832b090048f34974
nan568cd22e9e82dfc166d7dff1
568df3f0832b090048f34711nan
nan568e5a38b4a797c664143dda
```
I have tried this, also:
```
df3['feedback_request_id'] = ('' if df3['_id']=='nan' else df3['_id'].map(str)) + ('' if df3['feedback_id']=='nan' else df3['feedback_id'].map(str))
```
But it's giving the error:
```
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
```

Response:
You can use combine_first or fillna:
```
print df['feedback_id'].combine_first(df['_id'])
0    568a8c25cac4991645c287ac
1    568df45b177e30c6487d3603
2    568df434832b090048f34974
3    568cd22e9e82dfc166d7dff1
4    568df3f0832b090048f34711
5    568e5a38b4a797c664143dda
Name: feedback_id, dtype: object

print df['feedback_id'].fillna(df['_id'])
0    568a8c25cac4991645c287ac
1    568df45b177e30c6487d3603
2    568df434832b090048f34974
3    568cd22e9e82dfc166d7dff1
4    568df3f0832b090048f34711
5    568e5a38b4a797c664143dda
Name: feedback_id, dtype: object
```


Question:
Reverting from multiindex to single index dataframe in pandas
```
NI
YEAR MONTH datetime
2000 1     2000-01-01   NaN
           2000-01-02   NaN
           2000-01-03   NaN
           2000-01-04   NaN
           2000-01-05   NaN
```
In the dataframe above, I have a multilevel index consisting of the columns:
```
names=[u'YEAR', u'MONTH', u'datetime']
```
How do I revert to a dataframe with 'datetime' as index and 'YEAR' and 'MONTH' as normal columns?

Response:
pass level=[0,1] to just reset those levels:
```
dist_df = dist_df.reset_index(level=[0,1])

```python
df.reset_index(level=[0,1])
#Output
#YEAR  MONTH  NI
#datetime
#2000-01-01  2000      1   NaN
#2000-01-02  2000      1   NaN
#2000-01-03  2000      1   NaN
#2000-01-04  2000      1   NaN
#2000-01-05  2000      1   NaN
#```
#you can pass the label names alternatively:
#```
#df.reset_index(level=['YEAR','MONTH'])
#```
```


Question:
Pandas - Compute z-score for all columns I have a dataframe containing a single column of IDs and all other columns are numerical values for which I want to compute z-scores. Here's a subsection of it:
```none
ID      Age    BMI    Risk Factor
PT 6    48     19.3    4
PT 8    43     20.9    NaN
PT 2    39     18.1    3
PT 9    41     19.5    NaN
```
Some of my columns contain NaN values which I do not want to include into the z-score calculations so I intend to use a solution offered to this question: how to zscore normalize pandas column with nans?
```py
df['zscore'] = (df.a - df.a.mean())/df.a.std(ddof=0)
```
I'm interested in applying this solution to all of my columns except the ID column to produce a new dataframe which I can save as an Excel file using
```py
df2.to_excel("Z-Scores.xlsx")
```
So basically; how can I compute z-scores for each column (ignoring NaN values) and push everything into a new dataframe? SIDENOTE: there is a concept in pandas called "indexing" which intimidates me because I do not understand it well. If indexing is a crucial part of solving this problem, please dumb down your explanation of indexing.

Response:
Using Scipy's zscore function:
```
df = pd.DataFrame(np.random.randint(100, 200, size=(5, 3)), columns=['A', 'B', 'C'])
df

|    |   A |   B |   C |
|---:|----:|----:|----:|
|  0 | 163 | 163 | 159 |
|  1 | 120 | 153 | 181 |
|  2 | 130 | 199 | 108 |
|  3 | 108 | 188 | 157 |
|  4 | 109 | 171 | 119 |

from scipy.stats import zscore
df.apply(zscore)

|    |         A |         B |         C |
|---:|----------:|----------:|----------:|
|  0 |  1.83447  | -0.708023 |  0.523362 |
|  1 | -0.297482 | -1.30804  |  1.3342   |
|  2 |  0.198321 |  1.45205  | -1.35632  |
|  3 | -0.892446 |  0.792025 |  0.449649 |
|  4 | -0.842866 | -0.228007 | -0.950897 |
```
If not all the columns of your data frame are numeric, then you can apply the Z-score function only to the numeric columns using the select_dtypes function:
```
# Note that `select_dtypes` returns a data frame. We are selecting only the columns
numeric_cols = df.select_dtypes(include=[np.number]).columns
df[numeric_cols].apply(zscore)

|    |         A |         B |         C |
|---:|----------:|----------:|----------:|
|  0 |  1.83447  | -0.708023 |  0.523362 |
|  1 | -0.297482 | -1.30804  |  1.3342   |
|  2 |  0.198321 |  1.45205  | -1.35632  |
|  3 | -0.892446 |  0.792025 |  0.449649 |
|  4 | -0.842866 | -0.228007 | -0.950897 |
```


Question:
Accessing every 1st element of Pandas DataFrame column containing lists I have a Pandas DataFrame with a column containing lists objects
```
A
0   [1,2]
1   [3,4]
2   [8,9]
3   [2,6]
```
How can I access the first element of each list and save it into a new column of the DataFrame? To get a result like this:
```
A     new_col
0   [1,2]      1
1   [3,4]      3
2   [8,9]      8
3   [2,6]      2
```
I know this could be done via iterating over each row, but is there any "pythonic" way?

Response:
As always, remember that storing non-scalar objects in frames is generally disfavoured, and should really only be used as a temporary intermediate step. That said, you can use the .str accessor even though it's not a column of strings:
```

```python
df = pd.DataFrame({"A": [[1,2],[3,4],[8,9],[2,6]]})
```

```python
df["new_col"] = df["A"].str[0]
```

```python
df
#Output
#        A  new_col
#0  [1, 2]        1
#1  [3, 4]        3
#2  [8, 9]        8
#3  [2, 6]        2
```

```python
df["new_col"]
#Output
#0    1
#1    3
#2    8
#3    2
#Name: new_col, dtype: int64
#```
```


Question:
Add a sequential counter column on groups to a pandas dataframe I feel like there is a better way than this:
```py
import pandas as pd
df = pd.DataFrame(
    columns="   index    c1    c2    v1 ".split(),
    data= [
            [       0,  "A",  "X",    3, ],
            [       1,  "A",  "X",    5, ],
            [       2,  "A",  "Y",    7, ],
            [       3,  "A",  "Y",    1, ],
            [       4,  "B",  "X",    3, ],
            [       5,  "B",  "X",    1, ],
            [       6,  "B",  "X",    3, ],
            [       7,  "B",  "Y",    1, ],
            [       8,  "C",  "X",    7, ],
            [       9,  "C",  "Y",    4, ],
            [      10,  "C",  "Y",    1, ],
            [      11,  "C",  "Y",    6, ],]).set_index("index", drop=True)
def callback(x):
    x['seq'] = range(1, x.shape[0] + 1)
    return x
df = df.groupby(['c1', 'c2']).apply(callback)
print df
```
To achieve this:
```none
c1 c2  v1  seq
0   A  X   3    1
1   A  X   5    2
2   A  Y   7    1
3   A  Y   1    2
4   B  X   3    1
5   B  X   1    2
6   B  X   3    3
7   B  Y   1    1
8   C  X   7    1
9   C  Y   4    1
10  C  Y   1    2
11  C  Y   6    3
```
Is there a way to do it that avoids the callback?

Response:
use cumcount(), see docs here
```

```python
df.groupby(['c1', 'c2']).cumcount()
#Output
#0     0
#1     1
#2     0
#3     1
#4     0
#5     1
#6     2
#7     0
#8     0
#9     0
#10    1
#11    2
#dtype: int64
#```
#If you want orderings starting at 1
#```
```

```python
df.groupby(['c1', 'c2']).cumcount()+1
#Output
#0     1
#1     2
#2     1
#3     2
#4     1
#5     2
#6     3
#7     1
#8     1
#9     1
#10    2
#11    3
#dtype: int64
#```
```


Question:
Count frequency of values in pandas DataFrame column [duplicate] This question already has answers here: Count the frequency that a value occurs in a dataframe column (16 answers) Closed 7 months ago. I want to count number of times each values is appearing in dataframe. Here is my dataframe - df:
```
status
1     N
2     N
3     C
4     N
5     S
6     N
7     N
8     S
9     N
10    N
11    N
12    S
13    N
14    C
15    N
16    N
17    N
18    N
19    S
20    N
```
I want to dictionary of counts: ex. counts = {N: 14, C:2, S:4} I have tried df['status']['N'] but it gives keyError and also df['status'].value_counts but no use.

Response:
You can use value_counts and to_dict:
```
print df['status'].value_counts()
N    14
S     4
C     2
Name: status, dtype: int64

counts = df['status'].value_counts().to_dict()
print counts
{'S': 4, 'C': 2, 'N': 14}
```


Question:
Adding calculated column(s) to a dataframe in pandas I have an OHLC price data set, that I have parsed from CSV into a Pandas dataframe and resampled to 15 min bars:
```
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 500047 entries, 1998-05-04 04:45:00 to 2012-08-07 00:15:00
Freq: 15T
Data columns:
Close    363152  non-null values
High     363152  non-null values
Low      363152  non-null values
Open     363152  non-null values
dtypes: float64(4)
```
I would like to add various calculated columns, starting with simple ones such as period Range (H-L) and then booleans to indicate the occurrence of price patterns that I will define - e.g. a hammer candle pattern, for which a sample definition:
```
def closed_in_top_half_of_range(h,l,c):
    return c > l + (h-l)/2

def lower_wick(o,l,c):
    return min(o,c)-l

def real_body(o,c):
    return abs(c-o)

def lower_wick_at_least_twice_real_body(o,l,c):
    return lower_wick(o,l,c) >= 2 * real_body(o,c)

def is_hammer(row):
    return lower_wick_at_least_twice_real_body(row["Open"],row["Low"],row["Close"]) \
    and closed_in_top_half_of_range(row["High"],row["Low"],row["Close"])
```
Basic problem: how do I map the function to the column, specifically where I would like to reference more than one other column or the whole row or whatever? This post deals with adding two calculated columns off of a single source column, which is close, but not quite it. And slightly more advanced: for price patterns that are determined with reference to more than a single bar (T), how can I reference different rows (e.g. T-1, T-2 etc.) from within the function definition?

Response:
The exact code will vary for each of the columns you want to do, but it's likely you'll want to use the map and apply functions. In some cases you can just compute using the existing columns directly, since the columns are Pandas Series objects, which also work as Numpy arrays, which automatically work element-wise for usual mathematical operations.
```

```python
d
#Output
#    A   B  C
#0  11  13  5
#1   6   7  4
#2   8   3  6
#3   4   8  7
#4   0   1  7
```

```python
(d.A + d.B) / d.C
#Output
#0    4.800000
#1    3.250000
#2    1.833333
#3    1.714286
#4    0.142857
```

```python
d.A > d.C
#Output
#0     True
#1     True
#2     True
#3    False
#4    False
#```
#If you need to use operations like max and min within a row, you can use apply with axis=1 to apply any function you like to each row. Here's an example that computes min(A, B)-C, which seems to be like your "lower wick":
#```
```

```python
d.apply(lambda row: min([row['A'], row['B']])-row['C'], axis=1)
#Output
#0    6
#1    2
#2   -3
#3   -3
#4   -7
#```
#Hopefully that gives you some idea of how to proceed. Edit: to compare rows against neighboring rows, the simplest approach is to slice the columns you want to compare, leaving off the beginning/end, and then compare the resulting slices. For instance, this will tell you for which rows the element in column A is less than the next row's element in column C:
#```
#d['A'][:-1] < d['C'][1:]
#```
#and this does it the other way, telling you which rows have A less than the preceding row's C:
#```
#d['A'][1:] < d['C'][:-1]
#```
#Doing ['A"][:-1] slices off the last element of column A, and doing ['C'][1:] slices off the first element of column C, so when you line these two up and compare them, you're comparing each element in A with the C from the following row.
```


Question:
Diff of two Dataframes I need to compare two dataframes of different size row-wise and print out non matching rows. Lets take the following two:
```
df1 = DataFrame({
'Buyer': ['Carl', 'Carl', 'Carl'],
'Quantity': [18, 3, 5, ]})

df2 = DataFrame({
'Buyer': ['Carl', 'Mark', 'Carl', 'Carl'],
'Quantity': [2, 1, 18, 5]})
```
What is the most efficient way to row-wise over df2 and print out rows not in df1 e.g.
```
Buyer     Quantity
Carl         2
Mark         1
```
Important: I do not want to have row:
```
Buyer     Quantity
Carl         3
```
Included in the diff: I have already tried: Comparing two dataframes of different length row by row and adding columns for each row with equal value and Compare two DataFrames and output their differences side-by-side But these do not match with my problem.

Response:
merge the 2 dfs using method 'outer' and pass param indicator=True this will tell you whether the rows are present in both/left only/right only, you can then filter the merged df after:
```

```python
merged = df1.merge(df2, indicator=True, how='outer')
merged[merged['_merge'] == 'right_only']
#Output
#Buyer  Quantity      _merge
#3  Carl         2  right_only
#4  Mark         1  right_only
#```
```


Question:
Why does concatenation of DataFrames get exponentially slower? I have a function which processes a DataFrame, largely to process data into buckets create a binary matrix of features in a particular column using pd.get_dummies(df[col]). To avoid processing all of my data using this function at once (which goes out of memory and causes iPython to crash), I have broken the large DataFrame into chunks using:
```
chunks = (len(df) / 10000) + 1
df_list = np.array_split(df, chunks)
```
pd.get_dummies(df) will automatically create new columns based on the contents of df[col] and these are likely to differ for each df in df_list. After processing, I am concatenating the DataFrames back together using:
```
for i, df_chunk in enumerate(df_list):
    print "chunk", i
    [x, y] = preprocess_data(df_chunk)
    super_x = pd.concat([super_x, x], axis=0)
    super_y = pd.concat([super_y, y], axis=0)
    print datetime.datetime.utcnow()
```
The processing time of the first chunk is perfectly acceptable, however, it grows per chunk! This is not to do with the preprocess_data(df_chunk) as there is no reason for it to increase. Is this increase in time occurring as a result of the call to pd.concat()? Please see log below:
```
chunks 6
chunk 0
2016-04-08 00:22:17.728849
chunk 1
2016-04-08 00:22:42.387693
chunk 2
2016-04-08 00:23:43.124381
chunk 3
2016-04-08 00:25:30.249369
chunk 4
2016-04-08 00:28:11.922305
chunk 5
2016-04-08 00:32:00.357365
```
Is there a workaround to speed this up? I have 2900 chunks to process so any help is appreciated! Open to any other suggestions in Python!

Response:
Never call DataFrame.append or pd.concat inside a for-loop. It leads to quadratic copying. pd.concat returns a new DataFrame. Space has to be allocated for the new DataFrame, and data from the old DataFrames have to be copied into the new DataFrame. Consider the amount of copying required by this line inside the for-loop (assuming each x has size 1):
```
super_x = pd.concat([super_x, x], axis=0)

| iteration | size of old super_x | size of x | copying required |
|         0 |                   0 |         1 |                1 |
|         1 |                   1 |         1 |                2 |
|         2 |                   2 |         1 |                3 |
|       ... |                     |           |                  |
|       N-1 |                 N-1 |         1 |                N |
```
1 + 2 + 3 + ... + N = N(N+1)/2. So there is O(N**2) copies required to complete the loop. Now consider
```
super_x = []
for i, df_chunk in enumerate(df_list):
    [x, y] = preprocess_data(df_chunk)
    super_x.append(x)
super_x = pd.concat(super_x, axis=0)
```
Appending to a list is an O(1) operation and does not require copying. Now there is a single call to pd.concat after the loop is done. This call to pd.concat requires N copies to be made, since super_x contains N DataFrames of size 1. So when constructed this way, super_x requires O(N) copies.


Question:
Pandas equivalent of Oracle Lead/Lag function First I'm new to pandas, but I'm already falling in love with it. I'm trying to implement the equivalent of the Lag function from Oracle. Let's suppose you have this DataFrame:
```
Date                   Group      Data
2014-05-14 09:10:00        A         1
2014-05-14 09:20:00        A         2
2014-05-14 09:30:00        A         3
2014-05-14 09:40:00        A         4
2014-05-14 09:50:00        A         5
2014-05-14 10:00:00        B         1
2014-05-14 10:10:00        B         2
2014-05-14 10:20:00        B         3
2014-05-14 10:30:00        B         4
```
If this was an oracle database and I wanted to create a lag function grouped by the "Group" column and ordered by the Date I could easily use this function:
```
LAG(Data,1,NULL) OVER (PARTITION BY Group ORDER BY Date ASC) AS Data_lagged
```
This would result in the following Table:
```
Date                   Group     Data    Data lagged
2014-05-14 09:10:00        A        1           Null
2014-05-14 09:20:00        A        2            1
2014-05-14 09:30:00        A        3            2
2014-05-14 09:40:00        A        4            3
2014-05-14 09:50:00        A        5            4
2014-05-14 10:00:00        B        1           Null
2014-05-14 10:10:00        B        2            1
2014-05-14 10:20:00        B        3            2
2014-05-14 10:30:00        B        4            3
```
In pandas I can set the date to be an index and use the shift method:
```
db["Data_lagged"] = db.Data.shift(1)
```
The only issue is that this doesn't group by a column. Even if I set the two columns Date and Group as indexes, I would still get the "5" in the lagged column. Is there a way to implement the equivalent of the Lead and lag functions in Pandas?

Response:
You could perform a groupby/apply (shift) operation:
```

```python
df['Data_lagged'] = df.groupby(['Group'])['Data'].shift(1)
```

```python
df
#Output
#Date Group  Data  Data_lagged
#2014-05-14  09:10:00     A     1          NaN
#2014-05-14  09:20:00     A     2            1
#2014-05-14  09:30:00     A     3            2
#2014-05-14  09:40:00     A     4            3
#2014-05-14  09:50:00     A     5            4
#2014-05-14  10:00:00     B     1          NaN
#2014-05-14  10:10:00     B     2            1
#2014-05-14  10:20:00     B     3            2
#2014-05-14  10:30:00     B     4            3
#
#[9 rows x 4 columns]
#```
#To obtain the ORDER BY Date ASC effect, you must sort the DataFrame first:
#```
#df['Data_lagged'] = (df.sort_values(by=['Date'], ascending=True)
#                       .groupby(['Group'])['Data'].shift(1))
#```
```


Question:
add columns different length pandas I have a problem with adding columns in pandas. I have DataFrame, dimensional is nxk. And in process I wiil need add columns with dimensional mx1, where m = [1,n], but I don't know m. When I try do it:
```
df['Name column'] = data
# type(data) = list
```
result:
```
AssertionError: Length of values does not match length of index
```
Can I add columns with different length?

Response:
If you use accepted answer, you'll lose your column names, as shown in the accepted answer example, and described in the documentation (emphasis added): The resulting axis will be labeled 0, ..., n - 1. This is useful if you are concatenating objects where the concatenation axis does not have meaningful indexing information. It looks like column names ('Name column') are meaningful to the Original Poster / Original Question. To save column names, use pandas.concat, but don't ignore_index (default value of ignore_index is False; so you can omit that argument altogether). Continue to use axis=1:
```
import pandas

# Note these columns have 3 rows of values:
original = pandas.DataFrame({
    'Age':[10, 12, 13],
    'Gender':['M','F','M']
})

# Note this column has 4 rows of values:
additional = pandas.DataFrame({
    'Name': ['Nate A', 'Jessie A', 'Daniel H', 'John D']
})

new = pandas.concat([original, additional], axis=1)
# Identical:
# new = pandas.concat([original, additional], ignore_index=False, axis=1)

print(new.head())

#          Age        Gender        Name
#0          10             M      Nate A
#1          12             F    Jessie A
#2          13             M    Daniel H
#3         NaN           NaN      John D
```
Notice how John D does not have an Age or a Gender.


Question:
How to drop a specific column of csv file while reading it using pandas? I need to remove a column with label name at the time of loading a csv using pandas. I am reading csv as follows and want to add parameters inside it to do so. Thanks. pd.read_csv("sample.csv") I know this to do after reading csv:
```
df.drop('name', axis=1)
```

Response:
If you know the column names prior, you can do it by setting usecols parameter When you know which columns to use Suppose you have csv file with columns ["id", "name", "last_name"] and you want just ["name", "last_name"]. You can do it as below:
```
import pandas as pd
df = pd.read_csv("sample.csv", usecols=["name", "last_name"])
```
when you want first N columns If you don't know the column names but you want first N columns from dataframe. You can do it by
```
import pandas as pd
df = pd.read_csv("sample.csv", usecols=[i for i in range(n)])
```
Edit When you know name of the column to be dropped
```
# Read column names from file
cols = list(pd.read_csv("sample_data.csv", nrows=1))
print(cols)

# Use list comprehension to remove the unwanted column in **usecol**
df= pd.read_csv("sample_data.csv", usecols =[i for i in cols if i != "name"])
```


Question:
How can I replicate rows of a Pandas DataFrame? My pandas dataframe looks like this:
```
Person  ID   ZipCode   Gender
0  12345   882  38182     Female
1  32917   271  88172     Male
2  18273   552  90291     Female
```
I want to replicate every row 3 times and reset the index to get:
```
Person  ID   ZipCode   Gender
0  12345   882  38182     Female
1  12345   882  38182     Female
2  12345   882  38182     Female
3  32917   271  88172     Male
4  32917   271  88172     Male
5  32917   271  88172     Male
6  18273   552  90291     Female
7  18273   552  90291     Female
8  18273   552  90291     Female
```
I tried solutions such as:
```
pd.concat([df[:5]]*3, ignore_index=True)
```
which adds the rows to the end of the dataframe, instead of having 3 duplicate lines one after the other And:
```
df.reindex(np.repeat(df.index.values, df['ID']), method='ffill')
```
But none of them worked.

Response:
Solutions: Use np.repeat: Version 1: Try using np.repeat:
```
newdf = pd.DataFrame(np.repeat(df.values, 3, axis=0))
newdf.columns = df.columns
print(newdf)
```
The above code will output:
```
Person   ID ZipCode  Gender
0  12345  882   38182  Female
1  12345  882   38182  Female
2  12345  882   38182  Female
3  32917  271   88172    Male
4  32917  271   88172    Male
5  32917  271   88172    Male
6  18273  552   90291  Female
7  18273  552   90291  Female
8  18273  552   90291  Female
```
np.repeat repeats the values of df, 3 times. Then we add the columns with assigning new_df.columns = df.columns. Version 2: You could also assign the column names in the first line, like below:
```
newdf = pd.DataFrame(np.repeat(df.values, 3, axis=0), columns=df.columns)
print(newdf)
```
The above code will also output:
```
Person   ID ZipCode  Gender
0  12345  882   38182  Female
1  12345  882   38182  Female
2  12345  882   38182  Female
3  32917  271   88172    Male
4  32917  271   88172    Male
5  32917  271   88172    Male
6  18273  552   90291  Female
7  18273  552   90291  Female
8  18273  552   90291  Female
```
Version 3: You could shorten it with loc and only repeat the index, like below:
```
newdf = df.loc[np.repeat(df.index, 3)].reset_index(drop=True)
print(newdf)
```
The above code will also output:
```
Person   ID ZipCode  Gender
0  12345  882   38182  Female
1  12345  882   38182  Female
2  12345  882   38182  Female
3  32917  271   88172    Male
4  32917  271   88172    Male
5  32917  271   88172    Male
6  18273  552   90291  Female
7  18273  552   90291  Female
8  18273  552   90291  Female
```
I use reset_index to replace the index with monotonic indexes (0, 1, 2, 3, 4...). Without np.repeat: Version 4: You could use the built-in pd.Index.repeat function, like the below:
```
newdf = df.loc[df.index.repeat(3)].reset_index(drop=True)
print(newdf)
```
The above code will also output:
```
Person   ID ZipCode  Gender
0  12345  882   38182  Female
1  12345  882   38182  Female
2  12345  882   38182  Female
3  32917  271   88172    Male
4  32917  271   88172    Male
5  32917  271   88172    Male
6  18273  552   90291  Female
7  18273  552   90291  Female
8  18273  552   90291  Female
```
Remember to add reset_index to line-up the index. Version 5: Or by using concat with sort_index, like below:
```
newdf = pd.concat([df] * 3).sort_index().reset_index(drop=True)
print(newdf)
```
The above code will also output:
```
Person   ID ZipCode  Gender
0  12345  882   38182  Female
1  12345  882   38182  Female
2  12345  882   38182  Female
3  32917  271   88172    Male
4  32917  271   88172    Male
5  32917  271   88172    Male
6  18273  552   90291  Female
7  18273  552   90291  Female
8  18273  552   90291  Female
```
Version 6: You could also use loc with Python list multiplication and sorted, like below:
```
newdf = df.loc[sorted([*df.index] * 3)].reset_index(drop=True)
print(newdf)
```
The above code will also output:
```
Person   ID ZipCode  Gender
0  12345  882   38182  Female
1  12345  882   38182  Female
2  12345  882   38182  Female
3  32917  271   88172    Male
4  32917  271   88172    Male
5  32917  271   88172    Male
6  18273  552   90291  Female
7  18273  552   90291  Female
8  18273  552   90291  Female
```
Timings: Timing with the following code:
```
import timeit
import pandas as pd
import numpy as np

df = pd.DataFrame({'Person': {0: 12345, 1: 32917, 2: 18273}, 'ID': {0: 882, 1: 271, 2: 552}, 'ZipCode': {0: 38182, 1: 88172, 2: 90291}, 'Gender': {0: 'Female', 1: 'Male', 2: 'Female'}})

def version1():
    newdf = pd.DataFrame(np.repeat(df.values, 3, axis=0))
    newdf.columns = df.columns

def version2():
    newdf = pd.DataFrame(np.repeat(df.values, 3, axis=0), columns=df.columns)

def version3():
    newdf = df.loc[np.repeat(df.index, 3)].reset_index(drop=True)

def version4():
    newdf = df.loc[df.index.repeat(3)].reset_index(drop=True)

def version5():
    newdf = pd.concat([df] * 3).sort_index().reset_index(drop=True)

def version6():
    newdf = df.loc[sorted([*df.index] * 3)].reset_index(drop=True)

print('Version 1 Speed:', timeit.timeit('version1()', 'from __main__ import version1', number=20000))
print('Version 2 Speed:', timeit.timeit('version2()', 'from __main__ import version2', number=20000))
print('Version 3 Speed:', timeit.timeit('version3()', 'from __main__ import version3', number=20000))
print('Version 4 Speed:', timeit.timeit('version4()', 'from __main__ import version4', number=20000))
print('Version 5 Speed:', timeit.timeit('version5()', 'from __main__ import version5', number=20000))
print('Version 6 Speed:', timeit.timeit('version6()', 'from __main__ import version6', number=20000))
```
Output:
```
Version 1 Speed: 9.879425965991686
Version 2 Speed: 7.752138633004506
Version 3 Speed: 7.078321029010112
Version 4 Speed: 8.01169377300539
Version 5 Speed: 19.853051771002356
Version 6 Speed: 9.801617017001263
```
We can see that Versions 2 & 3 are faster than the others, the reason for this is because they both use the np.repeat function, and numpy functions are very fast because they are implemented with C. Version 3 wins against Version 2 marginally due to the usage of loc instead of DataFrame. Version 5 is significantly slower because of the functions concat and sort_index, since concat copies DataFrames quadratically, which takes longer time. Fastest Version: Version 3.


Question:
Pandas - Slice large dataframe into chunks I have a large dataframe (>3MM rows) that I'm trying to pass through a function (the one below is largely simplified), and I keep getting a Memory Error message. I think I'm passing too large of a dataframe into the function, so I'm trying to: 1) Slice the dataframe into smaller chunks (preferably sliced by AcctName) 2) Pass the dataframe into the function 3) Concatenate the dataframes back into one large dataframe
```
def trans_times_2(df):
    df['Double_Transaction'] = df['Transaction'] * 2

large_df
AcctName   Timestamp    Transaction
ABC        12/1         12.12
ABC        12/2         20.89
ABC        12/3         51.93
DEF        12/2         13.12
DEF        12/8          9.93
DEF        12/9         92.09
GHI        12/1         14.33
GHI        12/6         21.99
GHI        12/12        98.81
```
I know that my function works properly, since it will work on a smaller dataframe (e.g. 40,000 rows). I tried the following, but I was unsuccessful with concatenating the small dataframes back into one large dataframe.
```
def split_df(df):
    new_df = []
    AcctNames = df.AcctName.unique()
    DataFrameDict = {elem: pd.DataFrame for elem in AcctNames}
    key_list = [k for k in DataFrameDict.keys()]
    new_df = []
    for key in DataFrameDict.keys():
        DataFrameDict[key] = df[:][df.AcctNames == key]
        trans_times_2(DataFrameDict[key])
    rejoined_df = pd.concat(new_df)
```
How I envision the dataframes being split:
```
df1
AcctName   Timestamp    Transaction  Double_Transaction
ABC        12/1         12.12        24.24
ABC        12/2         20.89        41.78
ABC        12/3         51.93        103.86

df2
AcctName   Timestamp    Transaction  Double_Transaction
DEF        12/2         13.12        26.24
DEF        12/8          9.93        19.86
DEF        12/9         92.09        184.18

df3
AcctName   Timestamp    Transaction  Double_Transaction
GHI        12/1         14.33        28.66
GHI        12/6         21.99        43.98
GHI        12/12        98.81        197.62
```

Response:
You can use list comprehension to split your dataframe into smaller dataframes contained in a list.
```
n = 200000  #chunk row size
list_df = [df[i:i+n] for i in range(0,df.shape[0],n)]
```
Or use numpy array_split,
```
list_df = np.array_split(df, math.ceil(len(df)/n))
```
You can access the chunks with:
```
list_df[0]
list_df[1]
etc...
```
Then you can assemble it back into a one dataframe using pd.concat. By AcctName
```
list_df = []

for n,g in df.groupby('AcctName'):
    list_df.append(g)
```


Question: using time zone in pandas to_datetime I have time from epochs timestamps I use data.Time_req = pd.to_datetime(data.Time_req) But I get UTC time, I need +5:30 from the given time. How do I tell pandas to use 'IST' timezone or just 5hrs 30 mins further to the time it currently shows me. eg. 7 hrs should become 12:30 hrs and so on.

Response:
You can use tz_localize to set the timezone to UTC/+0000, and then tz_convert to add the timezone you want:
```
start = pd.to_datetime('2015-02-24')
rng = pd.date_range(start, periods=10)

df = pd.DataFrame({'Date': rng, 'a': range(10)})

df.Date = df.Date.dt.tz_localize('UTC').dt.tz_convert('Asia/Kolkata')
print (df)
                       Date  a
0 2015-02-24 05:30:00+05:30  0
1 2015-02-25 05:30:00+05:30  1
2 2015-02-26 05:30:00+05:30  2
3 2015-02-27 05:30:00+05:30  3
4 2015-02-28 05:30:00+05:30  4
5 2015-03-01 05:30:00+05:30  5
6 2015-03-02 05:30:00+05:30  6
7 2015-03-03 05:30:00+05:30  7
8 2015-03-04 05:30:00+05:30  8
9 2015-03-05 05:30:00+05:30  9
```
Working with time zones. If need add Timedelta only:
```
df.Date = df.Date + pd.Timedelta('05:30:00')
print (df)
                 Date  a
0 2015-02-24 05:30:00  0
1 2015-02-25 05:30:00  1
2 2015-02-26 05:30:00  2
3 2015-02-27 05:30:00  3
4 2015-02-28 05:30:00  4
5 2015-03-01 05:30:00  5
6 2015-03-02 05:30:00  6
7 2015-03-03 05:30:00  7
8 2015-03-04 05:30:00  8
9 2015-03-05 05:30:00  9
```
NOTE: Adding Timedelta will change the epoch timestamp associated with the datetime object. This may not be desired for many applications.


Question:
Pandas 'describe' is not returning summary of all columns I am running 'describe()' on a dataframe and getting summaries of only int columns (pandas 14.0). The documentation says that for object columns frequency of most common value, and additional statistics would be returned. What could be wrong? (no error message is returned by the way) Edit: I think it's how the function is set to behave on mixed column types in a dataframe. Although the documentation fails to mention it. Example code:
```
df_test = pd.DataFrame({'$a':[1,2], '$b': [10,20]})
df_test.dtypes
df_test.describe()
df_test['$a'] = df_test['$a'].astype(str)
df_test.describe()
df_test['$a'].describe()
df_test['$b'].describe()
```
My ugly work around in the meanwhile:
```
def my_df_describe(df):
    objects = []
    numerics = []
    for c in df:
        if (df[c].dtype == object):
            objects.append(c)
        else:
            numerics.append(c)

    return df[numerics].describe(), df[objects].describe()
```

Response:
As of pandas v15.0, use the parameter, DataFrame.describe(include = 'all') to get a summary of all the columns when the dataframe has mixed column types. The default behavior is to only provide a summary for the numerical columns. Example:
```

```python
df = pd.DataFrame({'$a':['a', 'b', 'c', 'd', 'a'], '$b': np.arange(5)})
df.describe(include = 'all')
#Output
#$a    $b
#count   5   5.000000
#unique  4   NaN
#top     a   NaN
#freq    2   NaN
#mean    NaN 2.000000
#std     NaN 1.581139
#min     NaN 0.000000
#25%     NaN 1.000000
#50%     NaN 2.000000
#75%     NaN 3.000000
#max     NaN 4.000000
#```
#The numerical columns will have NaNs for summary statistics pertaining to objects (strings) and vice versa. Summarizing only numerical or object columns To call describe() on just the numerical columns use describe(include = [np.number]) To call describe() on just the objects (strings) using describe(include = ['O']).
#```
```

```python
df.describe(include = [np.number])
#Output
#$b
#count   5.000000
#mean    2.000000
#std     1.581139
#min     0.000000
#25%     1.000000
#50%     2.000000
#75%     3.000000
#max     4.000000
```

```python
df.describe(include = ['O'])
#Output
#$a
#count   5
#unique  4
#top     a
#freq    2
#```
```


Question:
convert entire pandas dataframe to integers in pandas (0.17.0) [duplicate] This question already has answers here: Change column type in pandas (16 answers) Closed 1 year ago. My question is very similar to this one, but I need to convert my entire dataframe instead of just a series. The to_numeric function only works on one series at a time and is not a good replacement for the deprecated convert_objects command. Is there a way to get similar results to the convert_objects(convert_numeric=True) command in the new pandas release? Thank you Mike Müller for your example. df.apply(pd.to_numeric) works very well if the values can all be converted to integers. What if in my dataframe I had strings that could not be converted into integers? Example:
```
df = pd.DataFrame({'ints': ['3', '5'], 'Words': ['Kobe', 'Bryant']})
df.dtypes
Out[59]:
Words    object
ints     object
dtype: object
```
Then I could run the deprecated function and get:
```
df = df.convert_objects(convert_numeric=True)
df.dtypes
Out[60]:
Words    object
ints      int64
dtype: object
```
Running the apply command gives me errors, even with try and except handling.

Response:
All columns convertible You can apply the function to all columns:
```
df.apply(pd.to_numeric)
```
Example:
```

```python
df = pd.DataFrame({'a': ['1', '2'],
#Output
#                       'b': ['45.8', '73.9'],
#                       'c': [10.5, 3.7]})
```

```python
df.info()
#Output
#<class 'pandas.core.frame.DataFrame'>
#Int64Index: 2 entries, 0 to 1
#Data columns (total 3 columns):
#a    2 non-null object
#b    2 non-null object
#c    2 non-null float64
#dtypes: float64(1), object(2)
#memory usage: 64.0+ bytes
```

```python
df.apply(pd.to_numeric).info()
#Output
#<class 'pandas.core.frame.DataFrame'>
#Int64Index: 2 entries, 0 to 1
#Data columns (total 3 columns):
#a    2 non-null int64
#b    2 non-null float64
#c    2 non-null float64
#dtypes: float64(2), int64(1)
#memory usage: 64.0 bytes
#```
#Not all columns convertible pd.to_numeric has the keyword argument errors:
#```
#Signature: pd.to_numeric(arg, errors='raise')
#  Docstring:
#  Convert argument to a numeric type.
```

Parameters
----------
arg : list, tuple or array of objects, or Series
errors : {'ignore', 'raise', 'coerce'}, default 'raise'
    - If 'raise', then invalid parsing will raise an exception
    - If 'coerce', then invalid parsing will be set as NaN
    - If 'ignore', then invalid parsing will return the input
```
Setting it to ignore will return the column unchanged if it cannot be converted into a numeric type. As pointed out by Anton Protopopov, the most elegant way is to supply ignore as keyword argument to apply():
```

```python
df = pd.DataFrame({'ints': ['3', '5'], 'Words': ['Kobe', 'Bryant']})
```

```python
df.apply(pd.to_numeric, errors='ignore').info()
#Output
#<class 'pandas.core.frame.DataFrame'>
#Int64Index: 2 entries, 0 to 1
#Data columns (total 2 columns):
#Words    2 non-null object
#ints     2 non-null int64
#dtypes: int64(1), object(1)
#memory usage: 48.0+ bytes
#```
#My previously suggested way, using partial from the module functools, is more verbose:
#```
```

```python
from functools import partial
```

```python
df = pd.DataFrame({'ints': ['3', '5'],
#Output
#                       'Words': ['Kobe', 'Bryant']})
```

```python
df.apply(partial(pd.to_numeric, errors='ignore')).info()
#Output
#<class 'pandas.core.frame.DataFrame'>
#Int64Index: 2 entries, 0 to 1
#Data columns (total 2 columns):
#Words    2 non-null object
#ints     2 non-null int64
#dtypes: int64(1), object(1)
#memory usage: 48.0+ bytes
#```
```


Question:
Get weekday/day-of-week for Datetime column of DataFrame I have a DataFrame df like the following (excerpt, 'Timestamp' are the index):
```
Timestamp              Value
2012-06-01 00:00:00     100
2012-06-01 00:15:00     150
2012-06-01 00:30:00     120
2012-06-01 01:00:00     220
2012-06-01 01:15:00      80
...and so on.
```
I need a new column df['weekday'] with the respective weekday/day-of-week of the timestamps. How can I get this?

Response:
Use the new dt.dayofweek property:
```

```python
df['weekday'] = df['Timestamp'].dt.dayofweek
df
#Output
#Timestamp  Value  weekday
#0 2012-06-01 00:00:00    100        4
#1 2012-06-01 00:15:00    150        4
#2 2012-06-01 00:30:00    120        4
#3 2012-06-01 01:00:00    220        4
#4 2012-06-01 01:15:00     80        4
#```
#In the situation where the Timestamp is your index you need to reset the index and then call the dt.dayofweek property:
#```
```

```python
df = df.reset_index()
df['weekday'] = df['Timestamp'].dt.dayofweek
df
#Output
#Timestamp  Value  weekday
#0 2012-06-01 00:00:00    100        4
#1 2012-06-01 00:15:00    150        4
#2 2012-06-01 00:30:00    120        4
#3 2012-06-01 01:00:00    220        4
#4 2012-06-01 01:15:00     80        4
#```
#Strangely if you try to create a series from the index in order to not reset the index you get NaN values as does using the result of reset_index to call the dt.dayofweek property without assigning the result of reset_index back to the original df:
#```
```

```python
df['weekday'] = pd.Series(df.index).dt.dayofweek
df
#Output
#Value  weekday
#Timestamp
#2012-06-01 00:00:00    100      NaN
#2012-06-01 00:15:00    150      NaN
#2012-06-01 00:30:00    120      NaN
#2012-06-01 01:00:00    220      NaN
#2012-06-01 01:15:00     80      NaN
```

```python
df['weekday'] = df.reset_index()['Timestamp'].dt.dayofweek
df
#Output
#Value  weekday
#Timestamp
#2012-06-01 00:00:00    100      NaN
#2012-06-01 00:15:00    150      NaN
#2012-06-01 00:30:00    120      NaN
#2012-06-01 01:00:00    220      NaN
#2012-06-01 01:15:00     80      NaN
#```
#EDIT As pointed out to me by user @joris you can just access the weekday attribute of the index so the following will work and is more compact: df['Weekday'] = df.index.weekday
```


Question:
Drop multiple columns in pandas I am trying to drop multiple columns (column 2 and 70 in my data set, indexed as 1 and 69 respectively) by index number in a pandas data frame with the following code:
```py
df.drop([df.columns[[1, 69]]], axis=1, inplace=True)
```
I get the following error:
```none
TypeError: unhashable type: 'Index'
```
And in my code the [1, 69] is highlighted and says:
```none
Expected type 'Integral', got 'list[int]' instead
```
The following code does what I want in two lines of repetitive code (first dropping col index 69, then 1, and order does matter because dropping earlier columns changes the index of later columns).
```py
df.drop([df.columns[69]], axis=1, inplace=True)
df.drop([df.columns[1]], axis=1, inplace=True)
```
Is there a way to do this in one line similar to the first code snippet above?

Response:
You don't need to wrap it in a list with [..], just provide the subselection of the columns index:
```
df.drop(df.columns[[1, 69]], axis=1, inplace=True)
```
as the index object is already regarded as list-like.


Question:
How do I tell if a column in a pandas dataframe is of type datetime? How do I tell if a column is numerical? I am trying to filter the columns in a pandas dataframe based on whether they are of type date or not. I can figure out which ones are, but then would have to parse that output or manually select columns. I want to select date columns automatically. Here's what I have so far as an example - I'd want to only select the 'date_col' column in this case.
```
import pandas as pd
df = pd.DataFrame([['Feb-2017', 1, 2],
                   ['Mar-2017', 1, 2],
                   ['Apr-2017', 1, 2],
                   ['May-2017', 1, 2]],
                  columns=['date_str', 'col1', 'col2'])
df['date_col'] = pd.to_datetime(df['date_str'])
df.dtypes
```
Out:
```
date_str            object
col1                 int64
col2                 int64
date_col    datetime64[ns]
dtype: object
```

Response:
I just encountered this issue and found that @charlie-haley's answer isn't quite general enough for my use case. In particular np.datetime64 doesn't seem to match datetime64[ns, UTC].
```py
df['date_col'] = pd.to_datetime(df['date_str'], utc=True)
print(df.date_str.dtype)  # datetime64[ns, UTC]
```
You could also extend the list of dtypes to include other types, but that doesn't seem like a good solution for future compatability, so I ended up using the is_datetime64_any_dtype function from the pandas api instead. In:
```py
from pandas.api.types import is_datetime64_any_dtype as is_datetime

df[[column for column in df.columns if is_datetime(df[column])]]
```
Out:
```py
date_col
0 2017-02-01 00:00:00+00:00
1 2017-03-01 00:00:00+00:00
2 2017-04-01 00:00:00+00:00
3 2017-05-01 00:00:00+00:00
```


Question:
set difference for pandas A simple pandas question: Is there a drop_duplicates() functionality to drop every row involved in the duplication? An equivalent question is the following: Does pandas have a set difference for dataframes? For example:
```

```python
df1 = pd.DataFrame({'col1':[1,2,3], 'col2':[2,3,4]})
```

```python
df2 = pd.DataFrame({'col1':[4,2,5], 'col2':[6,3,5]})
```

```python
df1
#Output
#col1  col2
#0     1     2
#1     2     3
#2     3     4
```

```python
df2
#Output
#col1  col2
#0     4     6
#1     2     3
#2     5     5
#```
#so maybe something like df2.set_diff(df1) will produce this:
#```
#col1  col2
#0     4     6
#2     5     5
#```
#However, I don't want to rely on indexes because in my case, I have to deal with dataframes that have distinct indexes. By the way, I initially thought about an extension of the current drop_duplicates() method, but now I realize that the second approach using properties of set theory would be far more useful in general. Both approaches solve my current problem, though. Thanks!
```

Response:
Bit convoluted but if you want to totally ignore the index data. Convert the contents of the dataframes to sets of tuples containing the columns:
```
ds1 = set(map(tuple, df1.values))
ds2 = set(map(tuple, df2.values))
```
This step will get rid of any duplicates in the dataframes as well (index ignored)
```
set([(1, 2), (3, 4), (2, 3)])   # ds1
```
can then use set methods to find anything. Eg to find differences:
```
ds1.difference(ds2)
```
gives: set([(1, 2), (3, 4)]) can take that back to dataframe if needed. Note have to transform set to list 1st as set cannot be used to construct dataframe:
```
pd.DataFrame(list(ds1.difference(ds2)))
```


Question:
Start row index from 1 instead of zero without creating additional column in pandas I know that I can reset the indices like so
```py
df.reset_index(inplace=True)
```
but this will start the index from 0. I want to start it from 1. How do I do that without creating any extra columns and by keeping the index/reset_index functionality and options? I do not want to create a new dataframe, so inplace=True should still apply.

Response:
Just assign directly a new index array:
```
df.index = np.arange(1, len(df)+1)
```
Or if the index is already 0 based, just:
```
df.index += 1
```
Example:
```

```python
df = pd.DataFrame({'a': np.random.randn(5)})
df
#Output
#a
#0  0.443638
#1  0.037882
#2 -0.210275
#3 -0.344092
#4  0.997045
```

```python
df.index = np.arange(1, len(df)+1)
df
#Output
#a
#1  0.443638
#2  0.037882
#3 -0.210275
#4 -0.344092
#5  0.997045
#```
#TIMINGS For some reason I can't take timings on reset_index but the following are timings on a 100,000 row df:
#```
```

```python
%timeit df.index = df.index + 1
The slowest run took 6.45 times longer than the fastest. This could mean that an intermediate result is being cached
10000 loops, best of 3: 107 µs per loop
```

```python
%timeit df.index = np.arange(1, len(df)+1)
10000 loops, best of 3: 154 µs per loop
```
So without the timing for reset_index I can't say definitively, however it looks like just adding 1 to each index value will be faster if the index is already 0 based
```


Question:
Filter data frame based on index value in Python [duplicate] This question already has answers here: Filter pandas dataframe by list [duplicate] (7 answers) Closed 6 years ago. I have a data frame df with thousands of rows, and a sample is this:
```
Index           A   B   C   D   E   F
    EX-A.1.A.B-1A  18   7   2   2   9   8
    EX-A.1.A.B-1C   0   0   0   0   0   0
    EX-A.1.A.B-4A   6   4   8   6   1   1
    EX-A.1.A.B-4C   0   0   0   0   0   0
    EX-A.1.A.B-4F   0   0   0   0   0   0
```
I also have a list my_list = ["EX-A.1.A.B-1A","EX-A.1.A.B-4A","EX-A.1.A.B-4F"] and I want to filter the df based on this list, therefore I want to keep the rows for which the index value is in the list my_list. I tried this in order to create a new filtered df: Filter_df = df[df.index in my_list] and I get this error:
```
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().
```
Any ideas on how I could do this? Thanks

Response:
To check if row indices are in my_list:
```
Filter_df = df[df.index.isin(my_list)]
```


Question:
Get particular row as series from pandas dataframe How do we get a particular filtered row as series? Example dataframe:
```

```python
df = pd.DataFrame({'date': [20130101, 20130101, 20130102], 'location': ['a', 'a', 'c']})
```

```python
df
#Output
#       date location
#0  20130101        a
#1  20130101        a
#2  20130102        c
#```
#I need to select the row where location is c as a series. I tried:
#```
#row = df[df["location"] == "c"].head(1)  # gives a dataframe
#row = df.ix[df["location"] == "c"]       # also gives a dataframe with single row
#```
#In either cases I can't the row as series.
```

Response:
Use the squeeze function that will remove one dimension from the dataframe:
```
df[df["location"] == "c"].squeeze()
Out[5]:
date        20130102
location           c
Name: 2, dtype: object
```
DataFrame.squeeze method acts the same way of the squeeze argument of the read_csv function when set to True: if the resulting dataframe is a 1-len dataframe, i.e. it has only one dimension (a column or a row), then the object is squeezed down to the smaller dimension object. In your case, you get a Series object from the DataFrame. The same logic applies if you squeeze a Panel down to a DataFrame. squeeze is explicit in your code and shows clearly your intent to "cast down" the object in hands because its dimension can be projected to a smaller one. If the dataframe has more than one column or row, squeeze has no effect.


Question:
Store numpy.array in cells of a Pandas.DataFrame I have a dataframe in which I would like to store 'raw' numpy.array:
```
df['COL_ARRAY'] = df.apply(lambda r: np.array(do_something_with_r), axis=1)
```
but it seems that pandas tries to 'unpack' the numpy.array. Is there a workaround? Other than using a wrapper (see edit below)? I tried reduce=False with no success. EDIT This works, but I have to use the 'dummy' Data class to wrap around the array, which is unsatisfactory and not very elegant.
```
class Data:
    def __init__(self, v):
        self.v = v

meas = pd.read_excel(DATA_FILE)
meas['DATA'] = meas.apply(
    lambda r: Data(np.array(pd.read_csv(r['filename'])))),
    axis=1
)
```

Response:
Use a wrapper around the numpy array i.e. pass the numpy array as list
```
a = np.array([5, 6, 7, 8])
df = pd.DataFrame({"a": [a]})
```
Output:
```
a
0  [5, 6, 7, 8]
```
Or you can use apply(np.array) by creating the tuples i.e. if you have a dataframe
```
df = pd.DataFrame({'id': [1, 2, 3, 4],
                   'a': ['on', 'on', 'off', 'off'],
                   'b': ['on', 'off', 'on', 'off']})

df['new'] = df.apply(lambda r: tuple(r), axis=1).apply(np.array)
```
Output :
```
a    b  id            new
0   on   on   1    [on, on, 1]
1   on  off   2   [on, off, 2]
2  off   on   3   [off, on, 3]
3  off  off   4  [off, off, 4]
```
```
df['new'][0]
```
Output :
```
array(['on', 'on', '1'], dtype='<U2')
```


Question:
Get rows based on distinct values from one column How can I get the rows by distinct values in COL2? For example, I have the dataframe below:
```
COL1   COL2
a.com  22
b.com  45
c.com  34
e.com  45
f.com  56
g.com  22
h.com  45
```
I want to get the rows based on unique values in COL2:
```
COL1  COL2
a.com 22
b.com 45
c.com 34
f.com 56
```
So, how can I get that? I would appreciate it very much if anyone can provide any help.

Response:
Use drop_duplicates with specifying column COL2 for check duplicates:
```
df = df.drop_duplicates('COL2')
#same as
#df = df.drop_duplicates('COL2', keep='first')
print (df)
    COL1  COL2
0  a.com    22
1  b.com    45
2  c.com    34
4  f.com    56
```
You can also keep only last values:
```
df = df.drop_duplicates('COL2', keep='last')
print (df)
    COL1  COL2
2  c.com    34
4  f.com    56
5  g.com    22
6  h.com    45
```
Or remove all duplicates:
```
df = df.drop_duplicates('COL2', keep=False)
print (df)
    COL1  COL2
2  c.com    34
4  f.com    56
```


Question:
Pandas - convert strings to time without date I've read loads of SO answers but can't find a clear solution. I have this data in a df called day1 which represents hours:
```
1    10:53
2    12:17
3    14:46
4    16:36
5    18:39
6    20:31
7    22:28
Name: time, dtype: object>
```
I want to convert it into a time format. But when I do this: day1.time = pd.to_datetime(day1.time, format='H%:M%') The result includes today's date:
```
1   2015-09-03 10:53:00
2   2015-09-03 12:17:00
3   2015-09-03 14:46:00
4   2015-09-03 16:36:00
5   2015-09-03 18:39:00
6   2015-09-03 20:31:00
7   2015-09-03 22:28:00
Name: time, dtype: datetime64[ns]>
```
It seems the format argument isn't working - how do I get the time as shown here without the date? Update The following formats the time correctly, but somehow the column is still an object type. Why doesn't it convert to datetime64? day1['time'] = pd.to_datetime(day1['time'], format='%H:%M').dt.time
```
1    10:53:00
2    12:17:00
3    14:46:00
4    16:36:00
5    18:39:00
6    20:31:00
7    22:28:00
Name: time, dtype: object>
```

Response:
After performing the conversion you can use the datetime accessor dt to access just the hour or time component:
```

```python
df['hour'] = pd.to_datetime(df['time'], format='%H:%M').dt.hour
df
#Output
#time  hour
#index
#1      10:53    10
#2      12:17    12
#3      14:46    14
#4      16:36    16
#5      18:39    18
#6      20:31    20
#7      22:28    22
#```
#Also your format string H%:M% is malformed, it's likely to raise a ValueError: ':' is a bad directive in format 'H%:M%' Regarding your last comment the dtype is datetime.time not datetime:
#```
```

```python
df['time'].iloc[0]
#Output
#datetime.time(10, 53)
#```
```


Question:
Why use pandas.assign rather than simply initialize new column? I just discovered the assign method for pandas dataframes, and it looks nice and very similar to dplyr's mutate in R. However, I've always gotten by by just initializing a new column 'on the fly'. Is there a reason why assign is better? For instance (based on the example in the pandas documentation), to create a new column in a dataframe, I could just do this:
```
df = DataFrame({'A': range(1, 11), 'B': np.random.randn(10)})
df['ln_A'] = np.log(df['A'])
```
but the pandas.DataFrame.assign documentation recommends doing this:
```
df.assign(ln_A = lambda x: np.log(x.A))
# or
newcol = np.log(df['A'])
df.assign(ln_A=newcol)
```
Both methods return the same dataframe. In fact, the first method (my 'on the fly' assignment) is significantly faster (0.202 seconds for 1000 iterations) than the .assign method (0.353 seconds for 1000 iterations). So is there a reason I should stop using my old method in favour of df.assign?

Response:
The difference concerns whether you wish to modify an existing frame, or create a new frame while maintaining the original frame as it was. In particular, DataFrame.assign returns you a new object that has a copy of the original data with the requested changes ... the original frame remains unchanged. In your particular case:
```

```python
df = DataFrame({'A': range(1, 11), 'B': np.random.randn(10)})
#Output
#```
#Now suppose you wish to create a new frame in which A is everywhere 1 without destroying df. Then you could use .assign
#```
```

```python
new_df = df.assign(A=1)
#Output
#```
#If you do not wish to maintain the original values, then clearly df["A"] = 1 will be more appropriate. This also explains the speed difference, by necessity .assign must copy the data while [...] does not.
```


Question:
Pandas: resample timeseries with groupby Given the below pandas DataFrame:
```

```python
times = pd.to_datetime(pd.Series(['2014-08-25 21:00:00','2014-08-25 21:04:00',
                                            '2014-08-25 22:07:00','2014-08-25 22:09:00']))
          locations = ['HK', 'LDN', 'LDN', 'LDN']
          event = ['foo', 'bar', 'baz', 'qux']
          df = pd.DataFrame({'Location': locations,
                             'Event': event}, index=times)
          df
#Output
#Event Location
#          2014-08-25 21:00:00  foo   HK
#          2014-08-25 21:04:00  bar   LDN
#          2014-08-25 22:07:00  baz   LDN
#          2014-08-25 22:09:00  qux   LDN
#```
#I would like resample the data to aggregate it hourly by count while grouping by location to produce a data frame that looks like this:
#```
#Out[115]:
#                               HK    LDN
#          2014-08-25 21:00:00  1     1
#          2014-08-25 22:00:00  0     2
#```
#I've tried various combinations of resample() and groupby() but with no luck. How would I go about this?
```

Response:
In my original post, I suggested using pd.TimeGrouper. Nowadays, use pd.Grouper instead of pd.TimeGrouper. The syntax is largely the same, but TimeGrouper is now deprecated in favor of pd.Grouper. Moreover, while pd.TimeGrouper could only group by DatetimeIndex, pd.Grouper can group by datetime columns which you can specify through the key parameter. You could use a pd.Grouper to group the DatetimeIndex'ed DataFrame by hour:
```
grouper = df.groupby([pd.Grouper(freq='1H'), 'Location'])
```
use count to count the number of events in each group:
```
grouper['Event'].count()
#                      Location
# 2014-08-25 21:00:00  HK          1
#                      LDN         1
# 2014-08-25 22:00:00  LDN         2
# Name: Event, dtype: int64
```
use unstack to move the Location index level to a column level:
```
grouper['Event'].count().unstack()
# Out[49]:
# Location             HK  LDN
# 2014-08-25 21:00:00   1    1
# 2014-08-25 22:00:00 NaN    2
```
and then use fillna to change the NaNs into zeros. Putting it all together,
```
grouper = df.groupby([pd.Grouper(freq='1H'), 'Location'])
result = grouper['Event'].count().unstack('Location').fillna(0)
```
yields
```
Location             HK  LDN
2014-08-25 21:00:00   1    1
2014-08-25 22:00:00   0    2
```


Question:
Pandas convert a column of list to dummies I have a dataframe where one column is a list of groups each of my users belongs to. Something like:
```
index groups
0     ['a','b','c']
1     ['c']
2     ['b','c','e']
3     ['a','c']
4     ['b','e']
```
And what I would like to do is create a series of dummy columns to identify which groups each user belongs to in order to run some analyses
```
index  a   b   c   d   e
0      1   1   1   0   0
1      0   0   1   0   0
2      0   1   1   0   1
3      1   0   1   0   0
4      0   1   0   0   0

pd.get_dummies(df['groups'])
```
won't work because that just returns a column for each different list in my column. The solution needs to be efficient as the dataframe will contain 500,000+ rows.

Response:
Using s for your df['groups']:
```

```python
s = pd.Series({0: ['a', 'b', 'c'], 1:['c'], 2: ['b', 'c', 'e'], 3: ['a', 'c'], 4: ['b', 'e'] })
```

```python
s
#Output
#0    [a, b, c]
#1          [c]
#2    [b, c, e]
#3       [a, c]
#4       [b, e]
#dtype: object
#```
#This is a possible solution:
#```
```

```python
pd.get_dummies(s.explode()).groupby(level=0).sum()
#Output
#a  b  c  e
#0  1  1  1  0
#1  0  0  1  0
#2  0  1  1  1
#3  1  0  1  0
#4  0  1  0  1
#```
#The logic of this is: .explode() flattens the series of lists to a series of single values (with the index keeping track of the original row number) pd.get_dummies( ) creating the dummies .groupby(level=0).sum() for combining the different rows that should be one row (by summing up grouped by the index (level=0), i.e. the original row number)) If this will be efficient enough, I don't know, but in any case, if performance is important, storing lists in a dataframe is not a very good idea. Updates since original answer Since version 0.25, s.explode() can be used to flatten the Series of lists, instead of the original s.apply(pd.Series).stack() Since version 1.3.0, using the level keyword in aggregations is deprecated and will be removed from newer versions soon, so is recommended to use df.groupby(level=0).sum() instead of df.sum(level=0)
```


Question:
return max value from pandas dataframe as a whole, not based on column or rows I am trying to get the max value from a panda dataframe as a whole. I am not interested in what row or column it came from. I am just interested in a single max value within the DataFrame. Here is my DataFrame:
```
df = pd.DataFrame({'group1': ['a','a','a','b','b','b','c','c','d','d','d','d','d'],
                        'group2': ['c','c','d','d','d','e','f','f','e','d','d','d','e'],
                        'value1': [1.1,2,3,4,5,6,7,8,9,1,2,3,4],
                        'value2': [7.1,8,9,10,11,12,43,12,34,5,6,2,3]})
```
This is what it looks like:
```
group1 group2  value1  value2
0       a      c     1.1     7.1
1       a      c     2.0     8.0
2       a      d     3.0     9.0
3       b      d     4.0    10.0
4       b      d     5.0    11.0
5       b      e     6.0    12.0
6       c      f     7.0    43.0
7       c      f     8.0    12.0
8       d      e     9.0    34.0
9       d      d     1.0     5.0
10      d      d     2.0     6.0
11      d      d     3.0     2.0
12      d      e     4.0     3.0
```
Expected output:
```
43.0
```
I was under the assumption that df.max() would do this job but it returns a max value for each column but I am not interested in that. I need the max from an entire dataframe.

Response:
The max of all the values in the DataFrame can be obtained using df.to_numpy().max(), or for pandas < 0.24.0 we use df.values.max():
```

```python
df.to_numpy().max()
#Output
#'f'
#```
#The max is f rather than 43.0 since, in CPython2,
#```
```

```python
'f' > 43.0
#Output
#True
#```
#In CPython2, Objects of different types ... are ordered by their type names. So any str compares as greater than any int since 'str' > 'int'. In Python3, comparison of strings and ints raises a TypeError. To find the max value in the numeric columns only, use
#```
#df.select_dtypes(include=[np.number]).max()
#```
```


Question:
How to flatten a pandas dataframe with some columns as json? I have a dataframe df that loads data from a database. Most of the columns are json strings while some are even list of jsons. For example:
```
id     name     columnA                               columnB
1     John     {"dist": "600", "time": "0:12.10"}    [{"pos": "1st", "value": "500"},{"pos": "2nd", "value": "300"},{"pos": "3rd", "value": "200"}, {"pos": "total", "value": "1000"}]
2     Mike     {"dist": "600"}                       [{"pos": "1st", "value": "500"},{"pos": "2nd", "value": "300"},{"pos": "total", "value": "800"}]
...
```
As you can see, not all the rows have the same number of elements in the json strings for a column. What I need to do is keep the normal columns like id and name as it is and flatten the json columns like so:
```
id    name   columnA.dist   columnA.time   columnB.pos.1st   columnB.pos.2nd   columnB.pos.3rd     columnB.pos.total
1     John   600            0:12.10        500               300               200                 1000
2     Mark   600            NaN            500               300               Nan                 800
```
I have tried using json_normalize like so:
```
from pandas.io.json import json_normalize
json_normalize(df)
```
But there seems to be some problems with keyerror. What is the correct way of doing this?

Response:
Here's a solution using json_normalize() again by using a custom function to get the data in the correct format understood by json_normalize function.
```
import ast
from pandas.io.json import json_normalize

def only_dict(d):
    '''
    Convert json string representation of dictionary to a python dict
    '''
    return ast.literal_eval(d)

def list_of_dicts(ld):
    '''
    Create a mapping of the tuples formed after
    converting json strings of list to a python list
    '''
    return dict([(list(d.values())[1], list(d.values())[0]) for d in ast.literal_eval(ld)])

A = json_normalize(df['columnA'].apply(only_dict).tolist()).add_prefix('columnA.')
B = json_normalize(df['columnB'].apply(list_of_dicts).tolist()).add_prefix('columnB.pos.')
```
Finally, join the DFs on the common index to get:
```
df[['id', 'name']].join([A, B])
```
EDIT:- As per the comment by @MartijnPieters, the recommended way of decoding the json strings would be to use json.loads() which is much faster when compared to using ast.literal_eval() if you know that the data source is JSON.


Question:
Replace all occurrences of a string in a pandas dataframe (Python) I have a pandas dataframe with about 20 columns. It is possible to replace all occurrences of a string (here a newline) by manually writing all column names:
```
df['columnname1'] = df['columnname1'].str.replace("\n","<br>")
df['columnname2'] = df['columnname2'].str.replace("\n","<br>")
df['columnname3'] = df['columnname3'].str.replace("\n","<br>")
...
df['columnname20'] = df['columnname20'].str.replace("\n","<br>")
```
This unfortunately does not work:
```
df = df.replace("\n","<br>")
```
Is there any other, more elegant solution?

Response:
You can use replace and pass the strings to find/replace as dictionary keys/items:
```
df.replace({'\n': '<br>'}, regex=True)
```
For example:
```

```python
df = pd.DataFrame({'a': ['1\n', '2\n', '3'], 'b': ['4\n', '5', '6\n']})
```

```python
df
#Output
#   a    b
#0  1\n  4\n
#1  2\n  5
#2  3    6\n
```

```python
df.replace({'\n': '
'}, regex=True)
#Output
#   a      b
#0  1
  4

#1  2
  5
#2  3      6

#```
#Note that this method returns a new DataFrame instance by default (it does not modify the original), so you'll need to either reassign the output:
#```
#df = df.replace({'\n': '
'}, regex=True)
#```
#or specify inplace=True:
#```
#df.replace({'\n': '
'}, regex=True, inplace=True)
#```
```


Question:
Filter a pandas dataframe using values from a dict I need to filter a data frame with a dict, constructed with the key being the column name and the value being the value that I want to filter:
```
filter_v = {'A':1, 'B':0, 'C':'This is right'}
# this would be the normal approach
df[(df['A'] == 1) & (df['B'] ==0)& (df['C'] == 'This is right')]
```
But I want to do something on the lines
```
for column, value in filter_v.items():
    df[df[column] == value]
```
but this will filter the data frame several times, one value at a time, and not apply all filters at the same time. Is there a way to do it programmatically? EDIT: an example:
```
df1 = pd.DataFrame({'A':[1,0,1,1, np.nan], 'B':[1,1,1,0,1], 'C':['right','right','wrong','right', 'right'],'D':[1,2,2,3,4]})
filter_v = {'A':1, 'B':0, 'C':'right'}
df1.loc[df1[filter_v.keys()].isin(filter_v.values()).all(axis=1), :]
```
gives
```
A   B   C   D
0   1   1   right   1
1   0   1   right   2
3   1   0   right   3
```
but the expected result was
```
A   B   C   D
3   1   0   right   3
```
only the last one should be selected.

Response:
IIUC, you should be able to do something like this:
```

```python
df1.loc[(df1[list(filter_v)] == pd.Series(filter_v)).all(axis=1)]
#Output
#   A  B      C  D
#3  1  0  right  3
#```
#This works by making a Series to compare against:
#```
```

```python
pd.Series(filter_v)
#Output
#A        1
#B        0
#C    right
#dtype: object
#```
#Selecting the corresponding part of df1:
#```
```

```python
df1[list(filter_v)]
#Output
#    A      C  B
#0   1  right  1
#1   0  right  1
#2   1  wrong  1
#3   1  right  0
#4 NaN  right  1
#```
#Finding where they match:
#```
```

```python
df1[list(filter_v)] == pd.Series(filter_v)
#Output
#       A      B      C
#0   True  False   True
#1  False  False   True
#2   True  False  False
#3   True   True   True
#4  False  False   True
#```
#Finding where they all match:
#```
```

```python
(df1[list(filter_v)] == pd.Series(filter_v)).all(axis=1)
#Output
#0    False
#1    False
#2    False
#3     True
#4    False
#dtype: bool
#```
#And finally using this to index into df1:
#```
```

```python
df1.loc[(df1[list(filter_v)] == pd.Series(filter_v)).all(axis=1)]
#Output
#   A  B      C  D
#3  1  0  right  3
#```
```


Question:
Converting Pandas dataframe into Spark dataframe error I'm trying to convert Pandas DF into Spark one. DF head:
```
10000001,1,0,1,12:35,OK,10002,1,0,9,f,NA,24,24,0,3,9,0,0,1,1,0,0,4,543
10000001,2,0,1,12:36,OK,10002,1,0,9,f,NA,24,24,0,3,9,2,1,1,3,1,3,2,611
10000002,1,0,4,12:19,PA,10003,1,1,7,f,NA,74,74,0,2,15,2,0,2,3,1,2,2,691
```
Code:
```
dataset = pd.read_csv("data/AS/test_v2.csv")
sc = SparkContext(conf=conf)
sqlCtx = SQLContext(sc)
sdf = sqlCtx.createDataFrame(dataset)
```
And I got an error:
```
TypeError: Can not merge type <class 'pyspark.sql.types.StringType'> and <class 'pyspark.sql.types.DoubleType'>
```

Response:
I made this script, It worked for my 10 pandas Data frames
```
from pyspark.sql.types import *

# Auxiliar functions
def equivalent_type(f):
    if f == 'datetime64[ns]': return TimestampType()
    elif f == 'int64': return LongType()
    elif f == 'int32': return IntegerType()
    elif f == 'float64': return DoubleType()
    elif f == 'float32': return FloatType()
    else: return StringType()

def define_structure(string, format_type):
    try: typo = equivalent_type(format_type)
    except: typo = StringType()
    return StructField(string, typo)

# Given pandas dataframe, it will return a spark's dataframe.
def pandas_to_spark(pandas_df):
    columns = list(pandas_df.columns)
    types = list(pandas_df.dtypes)
    struct_list = []
    for column, typo in zip(columns, types):
      struct_list.append(define_structure(column, typo))
    p_schema = StructType(struct_list)
    return sqlContext.createDataFrame(pandas_df, p_schema)
```
You can see it also in this gist With this you just have to call spark_df = pandas_to_spark(pandas_df)


Question:
Eliminating all data over a given percentile I have a pandas DataFrame called data with a column called ms. I want to eliminate all the rows where data.ms is above the 95% percentile. For now, I'm doing this:
```
limit = data.ms.describe(90)['95%']
valid_data = data[data['ms'] < limit]
```
which works, but I want to generalize that to any percentile. What's the best way to do that?

Response:
Use the Series.quantile() method:
```

```python
cols = list('abc')
```

```python
df = DataFrame(randn(10, len(cols)), columns=cols)
```

```python
df.a.quantile(0.95)
#Output
#1.5776961953820687
#```
#To filter out rows of df where df.a is greater than or equal to the 95th percentile do:
#```
```

```python
df[df.a < df.a.quantile(.95)]
#Output
#a      b      c
#0 -1.044 -0.247 -1.149
#2  0.395  0.591  0.764
#3 -0.564 -2.059  0.232
#4 -0.707 -0.736 -1.345
#5  0.978 -0.099  0.521
#6 -0.974  0.272 -0.649
#7  1.228  0.619 -0.849
#8 -0.170  0.458 -0.515
#9  1.465  1.019  0.966
#```
```


Question:
Python Pandas iterate over rows and access column names I am trying to iterate over the rows of a Python Pandas dataframe. Within each row of the DataFrame, I am trying to refer to each value along a row by its column name. Here is what I have:
```
import numpy as np
import pandas as pd

df = pd.DataFrame(np.random.rand(10,4),columns=list('ABCD'))
print df
          A         B         C         D
0  0.351741  0.186022  0.238705  0.081457
1  0.950817  0.665594  0.671151  0.730102
2  0.727996  0.442725  0.658816  0.003515
3  0.155604  0.567044  0.943466  0.666576
4  0.056922  0.751562  0.135624  0.597252
5  0.577770  0.995546  0.984923  0.123392
6  0.121061  0.490894  0.134702  0.358296
7  0.895856  0.617628  0.722529  0.794110
8  0.611006  0.328815  0.395859  0.507364
9  0.616169  0.527488  0.186614  0.278792
```
I used this approach to iterate, but it is only giving me part of the solution - after selecting a row in each iteration, how do I access row elements by their column name? Here is what I am trying to do:
```
for row in df.iterrows():
    print row.loc[0,'A']
    print row.A
    print row.index()
```
My understanding is that the row is a Pandas series. But I have no way to index into the Series. Is it possible to use column names while simultaneously iterating over rows?

Response:
I also like itertuples()
```
for row in df.itertuples():
    print(row.A)
    print(row.Index)
```
since row is a named tuples, if you meant to access values on each row this should be MUCH faster speed run :
```
df = pd.DataFrame([x for x in range(1000*1000)], columns=['A'])
st=time.time()
for index, row in df.iterrows():
    row.A
print(time.time()-st)
45.05799984931946

st=time.time()
for row in df.itertuples():
    row.A
print(time.time() - st)
0.48400020599365234
```


Question:
df.unique() on whole DataFrame based on a column I have a DataFrame df filled with rows and columns where there are duplicate Id's:
```none
Index   Id   Type
0       a1   A
1       a2   A
2       b1   B
3       b3   B
4       a1   A
...
```
When I use:
```py
uniqueId = df["Id"].unique()
```
I get a list of unique IDs. How can I apply this filtering on the whole DataFrame such that it keeps the structure but that the duplicates (based on "Id") are removed?

Response:
It seems you need DataFrame.drop_duplicates with parameter subset which specify where are test duplicates:
```
#keep first duplicate value
df = df.drop_duplicates(subset=['Id'])
print (df)
       Id Type
Index
0      a1    A
1      a2    A
2      b1    B
3      b3    B
```
```
#keep last duplicate value
df = df.drop_duplicates(subset=['Id'], keep='last')
print (df)
       Id Type
Index
1      a2    A
2      b1    B
3      b3    B
4      a1    A
```
```
#remove all duplicate values
df = df.drop_duplicates(subset=['Id'], keep=False)
print (df)
       Id Type
Index
1      a2    A
2      b1    B
3      b3    B
```


Question:
Create a set from a series in pandas I have a dataframe extracted from Kaggle's San Fransico Salaries: https://www.kaggle.com/kaggle/sf-salaries and I wish to create a set of the values of a column, for instance 'Status'. This is what I have tried but it brings a list of all the records instead of the set (sf is how I name the data frame).
```
a=set(sf['Status'])
print a
```
According to this webpage, this should work. How to construct a set out of list items in python?

Response:
If you only need to get list of unique values, you can just use unique method. If you want to have Python's set, then do set(some_series)
```

```python
s = pd.Series([1, 2, 3, 1, 1, 4])
```

```python
s.unique()
#Output
#array([1, 2, 3, 4])
```

```python
set(s)
#Output
#{1, 2, 3, 4}
#```
#However, if you have DataFrame, just select series out of it ( some_data_frame[''] ).
```


Question:
Calculate weighted average using a pandas/dataframe I have the following table. I want to calculate a weighted average grouped by each date based on the formula below. I can do this using some standard conventional code, but assuming that this data is in a pandas dataframe, is there any easier way to achieve this rather than through iteration?
```
Date        ID      wt      value   w_avg
01/01/2012  100     0.50    60      0.791666667
01/01/2012  101     0.75    80
01/01/2012  102     1.00    100
01/02/2012  201     0.50    100     0.722222222
01/02/2012  202     1.00    80
```
01/01/2012 w_avg = 0.5 * ( 60/ sum(60,80,100)) + .75 * (80/ sum(60,80,100)) + 1.0 * (100/sum(60,80,100)) 01/02/2012 w_avg = 0.5 * ( 100/ sum(100,80)) + 1.0 * ( 80/ sum(100,80))

Response:
Let's first create the example pandas dataframe:
```

```python
import numpy as np
```

```python
import pandas as pd
```

```python
index = pd.Index(['01/01/2012','01/01/2012','01/01/2012','01/02/2012','01/02/2012'], name='Date')
```

```python
df = pd.DataFrame({'ID':[100,101,102,201,202],'wt':[.5,.75,1,.5,1],'value':[60,80,100,100,80]},index=index)
```
Then, the average of 'wt' weighted by 'value' and grouped by the index is obtained as:
```
```

```python
df.groupby(df.index).apply(lambda x: np.average(x.wt, weights=x.value))
#Output
#Date
#01/01/2012    0.791667
#01/02/2012    0.722222
#dtype: float64
#```
#Alternatively, one can also define a function:
#```
```

```python
def grouped_weighted_avg(values, weights, by):
   ...:     return (values * weights).groupby(by).sum() / weights.groupby(by).sum()
```

```python
grouped_weighted_avg(values=df.wt, weights=df.value, by=df.index)
#Output
#Date
#01/01/2012    0.791667
#01/02/2012    0.722222
#dtype: float64
#```
```


Question:
Use a loop to plot n charts Python I have a set of data that I load into python using a pandas dataframe. What I would like to do is create a loop that will print a plot for all the elements in their own frame, not all on one. My data is in an excel file structured in this fashion:
```
Index | DATE  | AMB CO 1 | AMB CO 2 |...|AMB CO_n | TOTAL
1     | 1/1/12|  14      | 33       |...|  236    | 1600
.     | ...   | ...      | ...      |...|  ...    | ...
.     | ...   | ...      | ...      |...|  ...    | ...
.     | ...   | ...      | ...      |...|  ...    | ...
n
```
This is what I have for code so far:
```
import pandas as pd
import matplotlib.pyplot as plt
ambdf = pd.read_excel('Ambulance.xlsx',
                      sheetname='Sheet2', index_col=0, na_values=['NA'])
print type(ambdf)
print ambdf
print ambdf['EAS']

amb_plot = plt.plot(ambdf['EAS'], linewidth=2)
plt.title('EAS Ambulance Numbers')
plt.xlabel('Month')
plt.ylabel('Count of Deliveries')
print amb_plot

for i in ambdf:
    print plt.plot(ambdf[i], linewidth = 2)
```
I am thinking of doing something like this:
```
for i in ambdf:
    ambdf_plot = plt.plot(ambdf, linewidth = 2)
```
The above was not remotely what i wanted and it stems from my unfamiliarity with Pandas, MatplotLib etc, looking at some documentation though to me it looks like matplotlib is not even needed (question 2) So A) How can I produce a plot of data for every column in my df and B) do I need to use matplotlib or should I just use pandas to do it all? Thank you,

Response:
Ok, so the easiest method to create several plots is this:
```
import matplotlib.pyplot as plt
x=[[1,2,3,4],[1,2,3,4],[1,2,3,4],[1,2,3,4]]
y=[[1,2,3,4],[1,2,3,4],[1,2,3,4],[1,2,3,4]]
for i in range(len(x)):
    plt.figure()
    plt.plot(x[i],y[i])
    # Show/save figure as desired.
    plt.show()
# Can show all four figures at once by calling plt.show() here, outside the loop.
#plt.show()
```
Note that you need to create a figure every time or pyplot will plot in the first one created. If you want to create several data series all you need to do is:
```
import matplotlib.pyplot as plt
plt.figure()
x=[[1,2,3,4],[1,2,3,4],[1,2,3,4],[1,2,3,4]]
y=[[1,2,3,4],[2,3,4,5],[3,4,5,6],[7,8,9,10]]
plt.plot(x[0],y[0],'r',x[1],y[1],'g',x[2],y[2],'b',x[3],y[3],'k')
```
You could automate it by having a list of colours like ['r','g','b','k'] and then just calling both entries in this list and corresponding data to be plotted in a loop if you wanted to. If you just want to programmatically add data series to one plot something like this will do it (no new figure is created each time so everything is plotted in the same figure):
```
import matplotlib.pyplot as plt
x=[[1,2,3,4],[1,2,3,4],[1,2,3,4],[1,2,3,4]]
y=[[1,2,3,4],[2,3,4,5],[3,4,5,6],[7,8,9,10]]
colours=['r','g','b','k']
plt.figure() # In this example, all the plots will be in one figure.
for i in range(len(x)):
    plt.plot(x[i],y[i],colours[i])
plt.show()
```
If anything matplotlib has a very good documentation page with plenty of examples. 17 Dec 2019: added plt.show() and plt.figure() calls to clarify this part of the story.


Question:
dataframe.describe() suppress scientific notation [duplicate] This question already has answers here: Suppressing scientific notation in pandas? [duplicate] (6 answers) Closed 7 years ago. How do I suppress scientific notation output from dataframe.describe():
```
contrib_df["AMNT"].describe()

count    1.979680e+05
mean     5.915134e+02
std      1.379618e+04
min     -1.750000e+05
25%      4.000000e+01
50%      1.000000e+02
75%      2.500000e+02
max      3.000000e+06
Name: AMNT, dtype: float64
```
My data is of type float64:
```
contrib_df["AMNT"].dtypes

dtype('float64')
```

Response:
For single column:
```
contrib_df["AMNT"].describe().apply(lambda x: format(x, 'f'))
```
For entire DataFrame (as suggested by @databyte )
```
df.describe().apply(lambda s: s.apply('{0:.5f}'.format))
```
For whole DataFrame (as suggested by @Jayen):
```
contrib_df.describe().apply(lambda s: s.apply(lambda x: format(x, 'g')))
```
As the function describe returns a data frame, what the above function does is, it simply formats each row to the regular format. I wrote this answer because I was having a though, in my mind, that was ** It's pointless to get the count of 95 as 95.00000e+01** Also in our regular format its easier to compare. Before applying the above function we were getting
```
count    9.500000e+01
mean     5.621943e+05
std      2.716369e+06
min      4.770000e+02
25%      2.118160e+05
50%      2.599960e+05
75%      3.121170e+05
max      2.670423e+07
Name: salary, dtype: float64
```
After applying, we get
```
count          95.000000
mean       562194.294737
std       2716369.154553
min           477.000000
25%        211816.000000
50%        259996.000000
75%        312117.000000
max      26704229.000000
Name: salary, dtype: object
```


Question:
Pandas GroupBy and select rows with the minimum value in a specific column I have a DataFrame with columns A, B, and C. For each value of A, I would like to select the row with the minimum value in column B. That is, from this:
```
df = pd.DataFrame({'A': [1, 1, 1, 2, 2, 2],
                   'B': [4, 5, 2, 7, 4, 6],
                   'C': [3, 4, 10, 2, 4, 6]})
    A   B   C
0   1   4   3
1   1   5   4
2   1   2   10
3   2   7   2
4   2   4   4
5   2   6   6
```
I would like to get:
```
A   B   C
0   1   2   10
1   2   4   4
```
For the moment I am grouping by column A, then creating a value that indicates to me the rows I will keep:
```
a = data.groupby('A').min()
a['A'] = a.index
to_keep = [str(x[0]) + str(x[1]) for x in a[['A', 'B']].values]
data['id'] = data['A'].astype(str) + data['B'].astype('str')
data[data['id'].isin(to_keep)]
```
I am sure that there is a much more straightforward way to do this. I have seen many answers here that use MultiIndex, which I would prefer to avoid. Thank you for your help.

Response:
I feel like you're overthinking this. Just use groupby and idxmin:
```
df.loc[df.groupby('A').B.idxmin()]

   A  B   C
2  1  2  10
4  2  4   4
```
```
df.loc[df.groupby('A').B.idxmin()].reset_index(drop=True)

   A  B   C
0  1  2  10
1  2  4   4
```


Question: Remove NaN/NULL columns in a Pandas dataframe? I have a dataFrame in pandas and several of the columns have all null values. Is there a built in function which will let me remove those columns?

Response:
Yes, dropna. See http://pandas.pydata.org/pandas-docs/stable/missing_data.html and the DataFrame.dropna docstring:
```
Definition: DataFrame.dropna(self, axis=0, how='any', thresh=None, subset=None)
Docstring:
Return object with labels on given axis omitted where alternately any
or all of the data are missing

Parameters
----------
axis : {0, 1}
how : {'any', 'all'}
    any : if any NA values are present, drop that label
    all : if all values are NA, drop that label
thresh : int, default None
    int value : require that many non-NA values
subset : array-like
    Labels along other axis to consider, e.g. if you are dropping rows
    these would be a list of columns to include

Returns
-------
dropped : DataFrame
```
The specific command to run would be:
```
df=df.dropna(axis=1,how='all')
```


Question:
Get the mean across multiple Pandas DataFrames I'm generating a number of dataframes with the same shape, and I want to compare them to one another. I want to be able to get the mean and median across the dataframes.
```
Source.0  Source.1  Source.2  Source.3
cluster
0        0.001182  0.184535  0.814230  0.000054
1        0.000001  0.160490  0.839508  0.000001
2        0.000001  0.173829  0.826114  0.000055
3        0.000432  0.180065  0.819502  0.000001
4        0.000152  0.157041  0.842694  0.000113
5        0.000183  0.174142  0.825674  0.000001
6        0.000001  0.151556  0.848405  0.000038
7        0.000771  0.177583  0.821645  0.000001
8        0.000001  0.202059  0.797939  0.000001
9        0.000025  0.189537  0.810410  0.000028
10       0.006142  0.003041  0.493912  0.496905
11       0.003739  0.002367  0.514216  0.479678
12       0.002334  0.001517  0.529041  0.467108
13       0.003458  0.000001  0.532265  0.464276
14       0.000405  0.005655  0.527576  0.466364
15       0.002557  0.003233  0.507954  0.486256
16       0.004161  0.000001  0.491271  0.504568
17       0.001364  0.001330  0.528311  0.468996
18       0.002886  0.000001  0.506392  0.490721
19       0.001823  0.002498  0.509620  0.486059

         Source.0  Source.1  Source.2  Source.3
cluster
0        0.000001  0.197108  0.802495  0.000396
1        0.000001  0.157860  0.842076  0.000063
2        0.094956  0.203057  0.701662  0.000325
3        0.000001  0.181948  0.817841  0.000210
4        0.000003  0.169680  0.830316  0.000001
5        0.000362  0.177194  0.822443  0.000001
6        0.000001  0.146807  0.852924  0.000268
7        0.001087  0.178994  0.819564  0.000354
8        0.000001  0.202182  0.797333  0.000485
9        0.000348  0.181399  0.818252  0.000001
10       0.003050  0.000247  0.506777  0.489926
11       0.004420  0.000001  0.513927  0.481652
12       0.006488  0.001396  0.527197  0.464919
13       0.001510  0.000001  0.525987  0.472502
14       0.000001  0.000001  0.520737  0.479261
15       0.000001  0.001765  0.515658  0.482575
16       0.000001  0.000001  0.492550  0.507448
17       0.002855  0.000199  0.526535  0.470411
18       0.000001  0.001952  0.498303  0.499744
19       0.001232  0.000001  0.506612  0.492155
```
Then I want to get the mean of these two dataframes. What is the easiest way to do this? Just to clarify I want to get the mean for each particular cell when the indexes and columns of all the dataframes are exactly the same. So in the example I gave, the average for [0,Source.0] would be (0.001182 + 0.000001) / 2 = 0.0005915.

Response:
Assuming the two dataframes have the same columns, you could just concatenate them and compute your summary stats on the concatenated frames:
```
import numpy as np
import pandas as pd

# some random data frames
df1 = pd.DataFrame(dict(x=np.random.randn(100), y=np.random.randint(0, 5, 100)))
df2 = pd.DataFrame(dict(x=np.random.randn(100), y=np.random.randint(0, 5, 100)))

# concatenate them
df_concat = pd.concat((df1, df2))

print df_concat.mean()
# x   -0.163044
# y    2.120000
# dtype: float64

print df_concat.median()
# x   -0.192037
# y    2.000000
# dtype: float64
```
Update If you want to compute stats across each set of rows with the same index in the two datasets, you can use .groupby() to group the data by row index, then apply the mean, median etc.:
```
by_row_index = df_concat.groupby(df_concat.index)
df_means = by_row_index.mean()

print df_means.head()
#           x    y
# 0 -0.850794  1.5
# 1  0.159038  1.5
# 2  0.083278  1.0
# 3 -0.540336  0.5
# 4  0.390954  3.5
```
This method will work even when your dataframes have unequal numbers of rows - if a particular row index is missing in one of the two dataframes, the mean/median will be computed on the single existing row.