Question:
Convert Select Columns in Pandas Dataframe to Numpy Array I would like to convert everything but the first column of a pandas dataframe into a numpy array. For some reason using the columns= parameter of DataFrame.to_matrix() is not working. df:
```
viz  a1_count  a1_mean     a1_std
0   n         3        2   0.816497
1   n         0      NaN        NaN
2   n         2       51  50.000000
```
I tried X=df.as_matrix(columns=[df[1:]]) but this yields an array of all NaNs

Response:
the easy way is the "values" property df.iloc[:,1:].values
```
a=df.iloc[:,1:]
b=df.iloc[:,1:].values

print(type(df))
print(type(a))
print(type(b))
```
so, you can get type
```
<class 'pandas.core.frame.DataFrame'>
<class 'pandas.core.frame.DataFrame'>
<class 'numpy.ndarray'>
```


Question: Adding two pandas dataframes I have two dataframes, both indexed by timeseries. I need to add the elements together to form a new dataframe, but only if the index and column are the same. If the item does not exist in one of the dataframes then it should be treated as a zero. I've tried using .add but this sums regardless of index and column. Also tried a simple combined_data = dataframe1 + dataframe2 but this give a NaN if both dataframes don't have the element. Any suggestions?

Response:
How about x.add(y, fill_value=0)?
```
import pandas as pd

df1 = pd.DataFrame([(1,2),(3,4),(5,6)], columns=['a','b'])
Out:
   a  b
0  1  2
1  3  4
2  5  6

df2 = pd.DataFrame([(100,200),(300,400),(500,600)], columns=['a','b'])
Out:
     a    b
0  100  200
1  300  400
2  500  600

df_add = df1.add(df2, fill_value=0)
Out:
     a    b
0  101  202
1  303  404
2  505  606
```


Question:
Splitting dataframe into multiple dataframes I have a very large dataframe (around 1 million rows) with data from an experiment (60 respondents). I would like to split the dataframe into 60 dataframes (a dataframe for each participant). In the dataframe, data, there is a variable called 'name', which is the unique code for each participant. I have tried the following, but nothing happens (or execution does not stop within an hour). What I intend to do is to split the data into smaller dataframes, and append these to a list (datalist):
```
import pandas as pd

def splitframe(data, name='name'):

    n = data[name][0]

    df = pd.DataFrame(columns=data.columns)

    datalist = []

    for i in range(len(data)):
        if data[name][i] == n:
            df = df.append(data.iloc[i])
        else:
            datalist.append(df)
            df = pd.DataFrame(columns=data.columns)
            n = data[name][i]
            df = df.append(data.iloc[i])

    return datalist
```
I do not get an error message, the script just seems to run forever! Is there a smart way to do it?

Response:
Can I ask why not just do it by slicing the data frame. Something like
```
#create some data with Names column
data = pd.DataFrame({'Names': ['Joe', 'John', 'Jasper', 'Jez'] *4, 'Ob1' : np.random.rand(16), 'Ob2' : np.random.rand(16)})

#create unique list of names
UniqueNames = data.Names.unique()

#create a data frame dictionary to store your data frames
DataFrameDict = {elem : pd.DataFrame() for elem in UniqueNames}

for key in DataFrameDict.keys():
    DataFrameDict[key] = data[:][data.Names == key]
```
Hey presto you have a dictionary of data frames just as (I think) you want them. Need to access one? Just enter
```
DataFrameDict['Joe']
```


Question:
Annotate bars with values on Pandas bar plots I was looking for a way to annotate my bars in a Pandas bar plot with the rounded numerical values from my DataFrame.
```

```python
df=pd.DataFrame({'A':np.random.rand(2),'B':np.random.rand(2)},index=['value1','value2'] )
```

```python
df
#Output
#                 A         B
#  value1  0.440922  0.911800
#  value2  0.588242  0.797366
#```
#I would like to get something like this: I tried with this code sample, but the annotations are all centered on the x ticks:
#```
```

```python
ax = df.plot(kind='bar')
```

```python
for idx, label in enumerate(list(df.index)):
#Output
#        for acc in df.columns:
#            value = np.round(df.ix[idx][acc],decimals=2)
#            ax.annotate(value,
#                        (idx, value),
#                         xytext=(0, 15),
#                         textcoords='offset points')
#```
```

Response:
You get it directly from the axes' patches:
```
for p in ax.patches:
    ax.annotate(str(p.get_height()), (p.get_x() * 1.005, p.get_height() * 1.005))
```
You'll want to tweak the string formatting and the offsets to get things centered, maybe use the width from p.get_width(), but that should get you started. It may not work with stacked bar plots unless you track the offsets somewhere.


Question:
Pandas date_range to generate monthly data at beginning of the month I'm trying to generate a date range of monthly data where the day is always at the beginning of the month:
```
pd.date_range(start='1/1/1980', end='11/1/1991', freq='M')
```
This generates 1/31/1980, 2/29/1980, and so on. Instead, I just want 1/1/1980, 2/1/1980,... I've seen other question ask about generating data that is always on a specific day of the month, with answers saying it wasn't possible, but beginning of month surely must be possible!

Response:
You can do this by changing the freq argument from 'M' to 'MS':
```
d = pandas.date_range(start='1/1/1980', end='11/1/1990', freq='MS')
print(d)
```
This should now print:
```
DatetimeIndex(['1980-01-01', '1980-02-01', '1980-03-01', '1980-04-01',
               '1980-05-01', '1980-06-01', '1980-07-01', '1980-08-01',
               '1980-09-01', '1980-10-01',
               ...
               '1990-02-01', '1990-03-01', '1990-04-01', '1990-05-01',
               '1990-06-01', '1990-07-01', '1990-08-01', '1990-09-01',
               '1990-10-01', '1990-11-01'],
              dtype='datetime64[ns]', length=131, freq='MS', tz=None)
```
Look into the offset aliases part of the documentation. There it states that 'M' is for the end of the month (month end frequency) while 'MS' for the beginning (month start frequency).


Question:
Combine pandas DataFrame query() method with isin() So I want to use isin() method with df.query(), to select rows with id in a list: id_list. Similar question was asked before, but they used typical df[df['id'].isin(id_list)] method. I'm wondering if there is a way to use df.query() instead.
```
df = pd.DataFrame({'a': list('aabbccddeeff'), 'b': list('aaaabbbbcccc'),
                   'c': np.random.randint(5, size=12),
                   'd': np.random.randint(9, size=12)})

id_list = ["a", "b", "c"]
```
And this yields an error
```
df.query('a == id_list')
```

Response:
You can also include the list within the query string:
```

```python
df.query('a in ["a", "b", "c"]')
#Output
#```
#This is the same as:
#```
```

```python
df.query('a in @id_list')
#Output
#```
```


Question:
T-test in Pandas If I want to calculate the mean of two categories in Pandas, I can do it like this:
```
data = {'Category': ['cat2','cat1','cat2','cat1','cat2','cat1','cat2','cat1','cat1','cat1','cat2'],
        'values': [1,2,3,1,2,3,1,2,3,5,1]}
my_data = DataFrame(data)
my_data.groupby('Category').mean()

Category:     values:
cat1     2.666667
cat2     1.600000
```
I have a lot of data formatted this way, and now I need to do a T-test to see if the mean of cat1 and cat2 are statistically different. How can I do that?

Response:
it depends what sort of t-test you want to do (one sided or two sided dependent or independent) but it should be as simple as:
```
from scipy.stats import ttest_ind

cat1 = my_data[my_data['Category']=='cat1']
cat2 = my_data[my_data['Category']=='cat2']

ttest_ind(cat1['values'], cat2['values'])

```python
(1.4927289925706944, 0.16970867501294376)
#Output
#```
#it returns a tuple with the t-statistic & the p-value see here for other t-tests http://docs.scipy.org/doc/scipy/reference/stats.html
```


Question:
Trying to remove commas and dollars signs with Pandas in Python Tring to remove the commas and dollars signs from the columns. But when I do, the table prints them out and still has them in there. Is there a different way to remove the commans and dollars signs using a pandas function. I was unuable to find anything in the API Docs or maybe i was looking in the wrong place
```
import pandas as pd
    import pandas_datareader.data as web

players = pd.read_html('http://www.usatoday.com/sports/mlb/salaries/2013/player/p/')

df1 = pd.DataFrame(players[0])

df1.drop(df1.columns[[0,3,4, 5, 6]], axis=1, inplace=True)
df1.columns = ['Player', 'Team', 'Avg_Annual']
df1['Avg_Annual'] = df1['Avg_Annual'].replace(',', '')

print (df1.head(10))
```

Response:
You have to access the str attribute per http://pandas.pydata.org/pandas-docs/stable/text.html
```
df1['Avg_Annual'] = df1['Avg_Annual'].str.replace(',', '')
df1['Avg_Annual'] = df1['Avg_Annual'].str.replace('$', '')
df1['Avg_Annual'] = df1['Avg_Annual'].astype(int)
```
alternately;
```
df1['Avg_Annual'] = df1['Avg_Annual'].str.replace(',', '').str.replace('$', '').astype(int)
```
if you want to prioritize time spent typing over readability.


Question:
Pandas: ValueError: cannot convert float NaN to integer I get ValueError: cannot convert float NaN to integer for following:
```
df = pandas.read_csv('zoom11.csv')
df[['x']] = df[['x']].astype(int)
```
The "x" is a column in the csv file, I cannot spot any float NaN in the file, and I don't understand the error or why I am getting it. When I read the column as String, then it has values like -1,0,1,...2000, all look very nice int numbers to me. When I read the column as float, then this can be loaded. Then it shows values as -1.0,0.0 etc, still there are no any NaN-s I tried with error_bad_lines = False and dtype parameter in read_csv to no avail. It just cancels loading with same exception. The file is not small (10+ M rows), so cannot inspect it manually, when I extract a small header part, then there is no error, but it happens with full file. So it is something in the file, but cannot detect what. Logically the csv should not have missing values, but even if there is some garbage then I would be ok to skip the rows. Or at least identify them, but I do not see way to scan through file and report conversion errors. Update: Using the hints in comments/answers I got my data clean with this:
```
# x contained NaN
df = df[~df['x'].isnull()]

# Y contained some other garbage, so null check was not enough
df = df[df['y'].str.isnumeric()]

# final conversion now worked
df[['x']] = df[['x']].astype(int)
df[['y']] = df[['y']].astype(int)
```

Response:
For identifying NaN values use boolean indexing:
```
print(df[df['x'].isnull()])
```
Then for removing all non-numeric values use to_numeric with parameter errors='coerce' - to replace non-numeric values to NaNs:
```
df['x'] = pd.to_numeric(df['x'], errors='coerce')
```
And for remove all rows with NaNs in column x use dropna:
```
df = df.dropna(subset=['x'])
```
Last convert values to ints:
```
df['x'] = df['x'].astype(int)
```


Question: How can I map True/False to 1/0 in a Pandas DataFrame? I have a column in python pandas DataFrame that has boolean True/False values, but for further calculations I need 1/0 representation. Is there a quick pandas/numpy way to do that?

Response:
A succinct way to convert a single column of boolean values to a column of integers 1 or 0:
```py
df["somecolumn"] = df["somecolumn"].astype(int)
```


Question:
Determining when a column value changes in pandas dataframe I am looking to write a quick script that will run through a csv file with two columns and provide me the rows in which the values in column B switch from one value to another: eg: dataframe:
```
# |  A  |  B
--+-----+-----
1 |  2  |  3
2 |  3  |  3
3 |  4  |  4
4 |  5  |  4
5 |  5  |  4
```
would tell me that the change happened between row 2 and row 3. I know how to get these values using for loops but I was hoping there was a more pythonic way of approaching this problem.

Response:
You can create a new column for the difference
```
> df['C'] = df['B'].diff()
> print df
   #  A  B   C
0  1  2  3 NaN
1  2  3  3   0
2  3  4  4   1
3  4  5  4   0
4  5  5  4   0

> df_filtered = df[df['C'] != 0]
> print df_filtered
   #  A  B  C
2  3  4  4  1
```
This will your required rows


Question:
How to one-hot-encode from a pandas column containing a list? I would like to break down a pandas column consisting of a list of elements into as many columns as there are unique elements i.e. one-hot-encode them (with value 1 representing a given element existing in a row and 0 in the case of absence). For example, taking dataframe df
```
Col1   Col2         Col3
 C      33     [Apple, Orange, Banana]
 A      2.5    [Apple, Grape]
 B      42     [Banana]
```
I would like to convert this to: df
```
Col1   Col2   Apple   Orange   Banana   Grape
 C      33     1        1        1       0
 A      2.5    1        0        0       1
 B      42     0        0        1       0
```
How can I use pandas/sklearn to achieve this?

Response:
We can also use sklearn.preprocessing.MultiLabelBinarizer: Often we want to use sparse DataFrame for the real world data in order to save a lot of RAM. Sparse solution (for Pandas v0.25.0+)
```
from sklearn.preprocessing import MultiLabelBinarizer

mlb = MultiLabelBinarizer(sparse_output=True)

df = df.join(
            pd.DataFrame.sparse.from_spmatrix(
                mlb.fit_transform(df.pop('Col3')),
                index=df.index,
                columns=mlb.classes_))
```
result:
```

```python
df
#Output
#Col1  Col2  Apple  Banana  Grape  Orange
#0    C  33.0      1       1      0       1
#1    A   2.5      1       0      1       0
#2    B  42.0      0       1      0       0
```

```python
df.dtypes
#Output
#Col1                object
#Col2               float64
#Apple     Sparse[int32, 0]
#Banana    Sparse[int32, 0]
#Grape     Sparse[int32, 0]
#Orange    Sparse[int32, 0]
#dtype: object
```

```python
df.memory_usage()
#Output
#Index     128
#Col1       24
#Col2       24
#Apple      16    #  <--- NOTE!
#Banana     16    #  <--- NOTE!
#Grape       8    #  <--- NOTE!
#Orange      8    #  <--- NOTE!
#dtype: int64
#```
#Dense solution
#```
#mlb = MultiLabelBinarizer()
#df = df.join(pd.DataFrame(mlb.fit_transform(df.pop('Col3')),
#                          columns=mlb.classes_,
#                          index=df.index))
#```
#Result:
#```
```

```python
df
#Output
#Col1  Col2  Apple  Banana  Grape  Orange
#0    C  33.0      1       1      0       1
#1    A   2.5      1       0      1       0
#2    B  42.0      0       1      0       0
#```
```


Question:
Pandas DataFrames with NaNs equality comparison In the context of unit testing some functions, I'm trying to establish the equality of 2 DataFrames using python pandas:
```
ipdb> expect
                            1   2
2012-01-01 00:00:00+00:00 NaN   3
2013-05-14 12:00:00+00:00   3 NaN

ipdb> df
identifier                  1   2
timestamp
2012-01-01 00:00:00+00:00 NaN   3
2013-05-14 12:00:00+00:00   3 NaN

ipdb> df[1][0]
nan

ipdb> df[1][0], expect[1][0]
(nan, nan)

ipdb> df[1][0] == expect[1][0]
False

ipdb> df[1][1] == expect[1][1]
True

ipdb> type(df[1][0])
<type 'numpy.float64'>

ipdb> type(expect[1][0])
<type 'numpy.float64'>

ipdb> (list(df[1]), list(expect[1]))
([nan, 3.0], [nan, 3.0])

ipdb> df1, df2 = (list(df[1]), list(expect[1])) ;; df1 == df2
False
```
Given that I'm trying to test the entire of expect against the entire of df, including NaN positions, what am I doing wrong? What is the simplest way to compare equality of Series/DataFrames including NaNs?

Response:
You can use assert_frame_equals with check_names=False (so as not to check the index/columns names), which will raise if they are not equal:
```

```python
from pandas.testing import assert_frame_equal
```

```python
assert_frame_equal(df, expected, check_names=False)
```
You can wrap this in a function with something like:
```
try:
    assert_frame_equal(df, expected, check_names=False)
    return True
except AssertionError:
    return False
```
In more recent pandas this functionality has been added as .equals:
```
df.equals(expected)
```
```


Question:
Remove rows in python less than a certain value I feel like this question must have been answered by someone before, but I can't find an answer on stack overflow! I have a dataframe result that looks like this and I want to remove all the values less than or equal to 10
```

```python
result
#Output
#                       Name              Value      Date
#189                   Sall                19.0  11/14/15
#191                     Sam               10.0  11/14/15
#192                 Richard               21.0  11/14/15
#193                  Ingrid                4.0  11/14/15
#```
#This command works and removes all the values that are 10:
#```
#df2 = result[result['Value'] != 10]
#```
#But when I try to add the <= qualifier I get the error message SyntaxError: invalid syntax
#```
#df3 = result[result['Value'] ! <= 10]
#```
#I feel like there is probably a really simple solution.
```

Response:
Instead of this
```
df3 = result[result['Value'] ! <= 10]
```
Use
```
df3 = result[~(result['Value'] <= 10)]
```
It will work. OR simply use
```
df3 = result[result['Value'] > 10]
```


Question:
Improve subplot size/spacing with many subplots I need to generate a whole bunch of vertically-stacked plots in matplotlib. The result will be saved using savefig and viewed on a webpage, so I don't care how tall the final image is, as long as the subplots are spaced so they don't overlap. No matter how big I allow the figure to be, the subplots always seem to overlap. My code currently looks like
```
import matplotlib.pyplot as plt
import my_other_module

titles, x_lists, y_lists = my_other_module.get_data()

fig = plt.figure(figsize=(10,60))
for i, y_list in enumerate(y_lists):
    plt.subplot(len(titles), 1, i)
    plt.xlabel("Some X label")
    plt.ylabel("Some Y label")
    plt.title(titles[i])
    plt.plot(x_lists[i],y_list)
fig.savefig('out.png', dpi=100)
```

Response:
Please review matplotlib: Tight Layout guide and try using matplotlib.pyplot.tight_layout, or matplotlib.figure.Figure.tight_layout As a quick example:
```
import matplotlib.pyplot as plt

fig, axes = plt.subplots(nrows=4, ncols=4, figsize=(8, 8))
fig.tight_layout() # Or equivalently,  "plt.tight_layout()"

plt.show()
```
Without Tight Layout With Tight Layout


Question:
How to convert list to row dataframe with Pandas I have a list of items like this:
```
A = ['1', 'd', 'p', 'bab', '']
```
My goal is to convert such list into a dataframe of 1 row and 5 columns. If I type pd.DataFrame(A) I get 5 rows and 1 column. What should I do in order to get the result I want?

Response:
You can use:
```
df = pd.DataFrame([A])
print (df)
   0  1  2    3 4
0  1  d  p  bab
```
If all values are strings:
```
df = pd.DataFrame(np.array(A).reshape(-1,len(A)))
print (df)

   0  1  2    3 4
0  1  d  p  bab
```
Thank you AKS:
```
df = pd.DataFrame(A).T
print (df)
   0  1  2    3 4
0  1  d  p  bab
```


Question:
What rules does Pandas use to generate a view vs a copy? I'm confused about the rules Pandas uses when deciding that a selection from a dataframe is a copy of the original dataframe, or a view on the original. If I have, for example,
```py
df = pd.DataFrame(np.random.randn(8,8), columns=list('ABCDEFGH'), index=range(1,9))
```
I understand that a query returns a copy so that something like
```py
foo = df.query('2 < index <= 5')
foo.loc[:,'E'] = 40
```
will have no effect on the original dataframe, df. I also understand that scalar or named slices return a view, so that assignments to these, such as
```py
df.iloc[3] = 70
```
or
```py
df.ix[1,'B':'E'] = 222
```
will change df. But I'm lost when it comes to more complicated cases. For example,
```py
df[df.C <= df.B] = 7654321
```
changes df, but
```py
df[df.C <= df.B].ix[:,'B':'E']
```
does not. Is there a simple rule that Pandas is using that I'm just missing? What's going on in these specific cases; and in particular, how do I change all values (or a subset of values) in a dataframe that satisfy a particular query (as I'm attempting to do in the last example above)? Note: This is not the same as this question; and I have read the documentation, but am not enlightened by it. I've also read through the "Related" questions on this topic, but I'm still missing the simple rule Pandas is using, and how I'd apply it to — for example — modify the values (or a subset of values) in a dataframe that satisfy a particular query.

Response:
Here's the rules, subsequent override: All operations generate a copy If inplace=True is provided, it will modify in-place; only some operations support this An indexer that sets, e.g. .loc/.iloc/.iat/.at will set inplace. An indexer that gets on a single-dtyped object is almost always a view (depending on the memory layout it may not be that's why this is not reliable). This is mainly for efficiency. (the example from above is for .query; this will always return a copy as its evaluated by numexpr) An indexer that gets on a multiple-dtyped object is always a copy. Your example of chained indexing
```
df[df.C <= df.B].loc[:,'B':'E']
```
is not guaranteed to work (and thus you should never do this). Instead do:
```
df.loc[df.C <= df.B, 'B':'E']
```
as this is faster and will always work The chained indexing is 2 separate python operations and thus cannot be reliably intercepted by pandas (you will oftentimes get a SettingWithCopyWarning, but that is not 100% detectable either). The dev docs, which you pointed, offer a much more full explanation.


Question:
shuffling/permutating a DataFrame in pandas What's a simple and efficient way to shuffle a dataframe in pandas, by rows or by columns? I.e. how to write a function shuffle(df, n, axis=0) that takes a dataframe, a number of shuffles n, and an axis (axis=0 is rows, axis=1 is columns) and returns a copy of the dataframe that has been shuffled n times. Edit: key is to do this without destroying the row/column labels of the dataframe. If you just shuffle df.index that loses all that information. I want the resulting df to be the same as the original except with the order of rows or order of columns different. Edit2: My question was unclear. When I say shuffle the rows, I mean shuffle each row independently. So if you have two columns a and b, I want each row shuffled on its own, so that you don't have the same associations between a and b as you do if you just re-order each row as a whole. Something like:
```
for 1...n:
  for each col in df: shuffle column
return new_df
```
But hopefully more efficient than naive looping. This does not work for me:
```
def shuffle(df, n, axis=0):
        shuffled_df = df.copy()
        for k in range(n):
            shuffled_df.apply(np.random.shuffle(shuffled_df.values),axis=axis)
        return shuffled_df

df = pandas.DataFrame({'A':range(10), 'B':range(10)})
shuffle(df, 5)
```

Response:
Use numpy's random.permuation function:
```

```python
df = pd.DataFrame({'A':range(10), 'B':range(10)})
```

```python
df
#Output
#A  B
#0  0  0
#1  1  1
#2  2  2
#3  3  3
#4  4  4
#5  5  5
#6  6  6
#7  7  7
#8  8  8
#9  9  9
```

```python
df.reindex(np.random.permutation(df.index))
#Output
#A  B
#0  0  0
#5  5  5
#6  6  6
#3  3  3
#8  8  8
#7  7  7
#9  9  9
#1  1  1
#2  2  2
#4  4  4
#```
```


Question:
Convert one DataFrame row to flat list I new to Python and I'm therefore having trouble converting a row in a DataFrame into a flat list. To do this I use the following code: Toy DataFrame:
```
import pandas as pd
d = {
     "a": [1, 2, 3, 4, 5],
     "b": [9, 8, 7, 6, 5],
     "n": ["a", "b", "c", "d", "e"]
}

df = pd.DataFrame(d)
```
My code:
```
df_note = df.loc[df.n == "d", ["a", "b"]].values #convert to array
df_note = df_note.tolist() #convert to nested list
df_note = reduce(lambda x, y: x + y, df_note) #convert to flat list
```
To me this code appears to be both gross and inefficient. The fact that I convert to an array before a list is what is causing the problem, i.e. the list to be nested. That withstanding, I can not find a means of converting the row directly to a list. Any advice? This question is not a dupe of this. In my case, I want the list to be flat.

Response:
You are almost there, actually just use flatten instead of reduce to unnest the array (instead of unnesting the list), and chain operations to have a one liner:
```
df.loc[df.n == "d", ['a','b']].values.flatten().tolist()
#[4, 6]
```


Question:
How to specify date format when using pandas.to_csv? The default output format of to_csv() is:
```
12/14/2012  12:00:00 AM
```
I cannot figure out how to output only the date part with specific format:
```
20121214
```
or date and time in two separate columns in the csv file:
```
20121214,  084530
```
The documentation is too brief to give me any clue as to how to do these. Can anyone help?

Response:
Since version v0.13.0 (January 3, 2014) of Pandas you can use the date_format parameter of the to_csv method:
```
df.to_csv(filename, date_format='%Y%m%d')
```


Question:
Summing over a multiindex level in a pandas series I would like to sum (marginalize) over one level in a series with a 3-level multiindex to produce a series with a 2 level multiindex. For example, if I have the following:
```py
ind = [tuple(x) for x in ['ABC', 'ABc', 'AbC', 'Abc', 'aBC', 'aBc', 'abC', 'abc']]
mi = pd.MultiIndex.from_tuples(ind)
data = pd.Series([264, 13, 29, 8, 152, 7, 15, 1], index=mi)

A  B  C    264
      c     13
   b  C     29
      c      8
a  B  C    152
      c      7
   b  C     15
      c      1
```
I would like to sum over the C variables to produce the following output:
```none
A  B    277
   b     37
a  B    159
   b     16
```
What is the best way in Pandas to do this?

Response:
If you know you always want to aggregate over the first two levels, then this is pretty easy:
```

```python
data.groupby(level=[0, 1]).sum()
#Output
#A  B    277
#   b     37
#a  B    159
#   b     16
#dtype: int64
#```
```


Question:
Pandas column bind (cbind) two data frames I've got a dataframe df_a with id information:
```none
unique_id lacet_number
15    5570613  TLA-0138365
24    5025490  EMP-0138757
36    4354431  DXN-0025343
```
and another dataframe df_b, with the same number of rows that I know correspond to the rows in df_a:
```none
latitude  longitude
0  -93.193560  31.217029
1  -93.948082  35.360874
2 -103.131508  37.787609
```
What I want to do is simply concatenate the two horizontally (similar to cbind in R) and get:
```none
unique_id lacet_number      latitude  longitude
0     5570613  TLA-0138365    -93.193560  31.217029
1     5025490  EMP-0138757    -93.948082  35.360874
2     4354431  DXN-0025343   -103.131508  37.787609
```
What I have tried:
```py
df_c = pd.concat([df_a, df_b], axis=1)
```
which gives me an outer join.
```none
unique_id lacet_number    latitude  longitude
0         NaN          NaN  -93.193560  31.217029
1         NaN          NaN  -93.948082  35.360874
2         NaN          NaN -103.131508  37.787609
15    5570613  TLA-0138365         NaN        NaN
24    5025490  EMP-0138757         NaN        NaN
36    4354431  DXN-0025343         NaN        NaN
```
The problem is that the indices for the two dataframes do not match. I read the documentation for pandas.concat, and saw that there is an option ignore_index. But that only applies to the concatenation axis, in my case the columns and it certainly is not the right choice for me. So my question is: is there a simple way to achieve this?

Response:
If you're sure the index row values are the same then to avoid the index alignment order then just call reset_index(), this will reset your index values back to start from 0:
```
df_c = pd.concat([df_a.reset_index(drop=True), df_b], axis=1)
```


Question:
Seaborn Bar Plot Ordering I have a pandas dataframe that has two columns. I need the plot ordered by the "Count" Column.
```
dicti=({'37':99943,'25':47228,'36':16933,'40':14996,'35':11791,'34':8030,'24' : 6319 ,'2'  :5055 ,'39' :4758 ,'38' :4611  })
pd_df = pd.DataFrame(list(dicti.iteritems()))
pd_df.columns =["Dim","Count"]
plt.figure(figsize=(12,8))
ax = sns.barplot(x="Dim", y= "Count",data=pd_df )
ax.get_yaxis().set_major_formatter(plt.FuncFormatter(lambda x, loc: "
{:,}".format(int(x))))
ax.set(xlabel="Dim", ylabel='Count')
for item in ax.get_xticklabels():
    item.set_rotation(90)
for i, v in enumerate(pd_df["Count"].iteritems()):
    ax.text(i ,v[1], "{:,}".format(v[1]), color='m', va ='bottom',
    rotation=45)
plt.tight_layout()
```
Right now the plot is getting ordered by the "Dim" column, I need it ordered by the "Count" column,How can I do this?

Response:
you can use the order parameter for this.
```
sns.barplot(x='Id', y="Speed", data=df, order=result['Id'])
```
Credits to Wayne. See the rest of his code. This link is still working for me. But, for the sake of convenience, I'm pasting the author's code here.
```
result = df.groupby(["Id"])['Speed'].aggregate(np.median).reset_index().sort_values('Speed')
sns.barplot(x='Id', y="Speed", data=df, order=result['Id'])
plt.show()
```
df
```
Id  Speed
0   1     30
1   1     35
2   1     31
3   2     20
4   2     25
```
result
```
Id   Speed
1   2   22.5
0   1   31.0
2   3   80.0
```


Question:
How to concatenate two dataframes without duplicates? I'd like to concatenate two dataframes A, B to a new one without duplicate rows (if rows in B already exist in A, don't add): Dataframe A:
```
I    II
0  1    2
1  3    1
```
Dataframe B:
```
I    II
0  5    6
1  3    1
```
New Dataframe:
```
I    II
0  1    2
1  3    1
2  5    6
```
How can I do this?

Response:
The simplest way is to just do the concatenation, and then drop duplicates.
```

```python
df1
#Output
#   A  B
#0  1  2
#1  3  1
```

```python
df2
#Output
#   A  B
#0  5  6
#1  3  1
```

```python
pandas.concat([df1,df2]).drop_duplicates().reset_index(drop=True)
#Output
#   A  B
#0  1  2
#1  3  1
#2  5  6
#```
#The reset_index(drop=True) is to fix up the index after the concat() and drop_duplicates(). Without it you will have an index of [0,1,0] instead of [0,1,2]. This could cause problems for further operations on this dataframe down the road if it isn't reset right away.
```


Question: List Highest Correlation Pairs from a Large Correlation Matrix in Pandas? How do you find the top correlations in a correlation matrix with Pandas? There are many answers on how to do this with R (Show correlations as an ordered list, not as a large matrix or Efficient way to get highly correlated pairs from large data set in Python or R), but I am wondering how to do it with pandas? In my case the matrix is 4460x4460, so can't do it visually.

Response:
You can use DataFrame.values to get an numpy array of the data and then use NumPy functions such as argsort() to get the most correlated pairs. But if you want to do this in pandas, you can unstack and sort the DataFrame:
```
import pandas as pd
import numpy as np

shape = (50, 4460)

data = np.random.normal(size=shape)

data[:, 1000] += data[:, 2000]

df = pd.DataFrame(data)

c = df.corr().abs()

s = c.unstack()
so = s.sort_values(kind="quicksort")

print so[-4470:-4460]
```
Here is the output:
```
2192  1522    0.636198
1522  2192    0.636198
3677  2027    0.641817
2027  3677    0.641817
242   130     0.646760
130   242     0.646760
1171  2733    0.670048
2733  1171    0.670048
1000  2000    0.742340
2000  1000    0.742340
dtype: float64
```


Question:
append dictionary to data frame I have a function, which returns a dictionary like this:
```
{'truth': 185.179993, 'day1': 197.22307753038834, 'day2': 197.26118010160317, 'day3': 197.19846975345905, 'day4': 197.1490578795196, 'day5': 197.37179265011116}
```
I am trying to append this dictionary to a dataframe like so:
```
output = pd.DataFrame()
output.append(dictionary, ignore_index=True)
print(output.head())
```
Unfortunately, the printing of the dataframe results in an empty dataframe. Any ideas?

Response:
Doesn't work starting pandas==2.0.0, deprecated since 1.4 You don't assign the value to the result.
```
output = pd.DataFrame()
output = output.append(dictionary, ignore_index=True)
print(output.head())
```


Question: Difference between groupby and pivot_table for pandas dataframes I just started learning Pandas and was wondering if there is any difference between groupby and pivot_table functions. Can anyone help me understand the difference between them?

Response:
Both pivot_table and groupby are used to aggregate your dataframe. The difference is only with regard to the shape of the result. Using pd.pivot_table(df, index=["a"], columns=["b"], values=["c"], aggfunc=np.sum) a table is created where a is on the row axis, b is on the column axis, and the values are the sum of c. Example:
```
df = pd.DataFrame({"a": [1,2,3,1,2,3], "b":[1,1,1,2,2,2], "c":np.random.rand(6)})
pd.pivot_table(df, index=["a"], columns=["b"], values=["c"], aggfunc=np.sum)

b         1         2
a
1  0.528470  0.484766
2  0.187277  0.144326
3  0.866832  0.650100
```
Using groupby, the dimensions given are placed into columns, and rows are created for each combination of those dimensions. In this example, we create a series of the sum of values c, grouped by all unique combinations of a and b.
```
df.groupby(['a','b'])['c'].sum()

a  b
1  1    0.528470
   2    0.484766
2  1    0.187277
   2    0.144326
3  1    0.866832
   2    0.650100
Name: c, dtype: float64
```
A similar usage of groupby is if we omit the ['c']. In this case, it creates a dataframe (not a series) of the sums of all remaining columns grouped by unique values of a and b.
```
print df.groupby(["a","b"]).sum()
            c
a b
1 1  0.528470
  2  0.484766
2 1  0.187277
  2  0.144326
3 1  0.866832
  2  0.650100
```


Question:
pandas.read_csv: how to skip comment lines I think I misunderstand the intention of read_csv. If I have a file 'j' like
```
# notes
a,b,c
# more notes
1,2,3
```
How can I pandas.read_csv this file, skipping any '#' commented lines? I see in the help 'comment' of lines is not supported but it indicates an empty line should be returned. I see an error
```
df = pandas.read_csv('j', comment='#')
```
CParserError: Error tokenizing data. C error: Expected 1 fields in line 2, saw 3 I'm currently on
```

```python
pandas.__version__
#Output
#'0.12.0rc1'
#```
#On version'0.12.0-199-g4c8ad82':
#```
```

```python
df = pandas.read_csv('j', comment='#', header=None)
```
CParserError: Error tokenizing data. C error: Expected 1 fields in line 2, saw 3
```

Response: So I believe in the latest releases of pandas (version 0.16.0), you could throw in the comment='#' parameter into pd.read_csv and this should skip commented out lines. These github issues shows that you can do this: https://github.com/pydata/pandas/issues/10548 https://github.com/pydata/pandas/issues/4623 See the documentation on read_csv: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html


Question:
Why isn't my Pandas 'apply' function referencing multiple columns working? [closed] Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers. This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers. Closed 6 years ago. Improve this question I have some problems with the Pandas apply function, when using multiple columns with the following dataframe
```
df = DataFrame ({'a' : np.random.randn(6),
                 'b' : ['foo', 'bar'] * 3,
                 'c' : np.random.randn(6)})
```
and the following function
```
def my_test(a, b):
    return a % b
```
When I try to apply this function with :
```
df['Value'] = df.apply(lambda row: my_test(row[a], row[c]), axis=1)
```
I get the error message:
```
NameError: ("global name 'a' is not defined", u'occurred at index 0')
```
I do not understand this message, I defined the name properly. I would highly appreciate any help on this issue Update Thanks for your help. I made indeed some syntax mistakes with the code, the index should be put ''. However I still get the same issue using a more complex function such as:
```
def my_test(a):
    cum_diff = 0
    for ix in df.index():
        cum_diff = cum_diff + (a - df['a'][ix])
    return cum_diff
```

Response:
Seems you forgot the '' of your string.
```

```python
df['Value'] = df.apply(lambda row: my_test(row['a'], row['c']), axis=1)
```

```python
df
#Output
#a    b         c     Value
#          0 -1.674308  foo  0.343801  0.044698
#          1 -2.163236  bar -2.046438 -0.116798
#          2 -0.199115  foo -0.458050 -0.199115
#          3  0.918646  bar -0.007185 -0.001006
#          4  1.336830  foo  0.534292  0.268245
#          5  0.976844  bar -0.773630 -0.570417
#```
#BTW, in my opinion, following way is more elegant:
#```
```

```python
def my_test2(row):
....:     return row['a'] % row['c']
....:
```

```python
df['Value'] = df.apply(my_test2, axis=1)
```
```


Question:
How to select the last column of dataframe I have done some searching for the answer to this question, but all I can figure out is this:
```
df[df.columns[len(df.columns)-1]]
```
which to me seems unweildy, and un-pythonic (and slow?). What is the easiest way to select the data for the last column in a pandas dataframe without specifying the name of the column?

Response:
Use iloc and select all rows (:) against the last column (-1):
```
df.iloc[:,-1:]
```


Question: Square of each element of a column in pandas How can I square each element of a column/series of a DataFrame in pandas (and create another column to hold the result)?

Response:
```

```python
import pandas as pd
```

```python
df = pd.DataFrame([[1,2],[3,4]], columns=list('ab'))
```

```python
df
#Output
#   a  b
#0  1  2
#1  3  4
```

```python
df['c'] = df['b']**2
```

```python
df
#Output
#   a  b   c
#0  1  2   4
#1  3  4  16
#```
```


Question:
Python: Removing Rows on Count condition [duplicate] This question already has answers here: Pandas equivalent of GROUP BY HAVING in SQL (3 answers) Closed last year. I have a problem filtering a pandas dataframe.
```
city
NYC
NYC
NYC
NYC
SYD
SYD
SEL
SEL
...

df.city.value_counts()
```
I would like to remove rows of cities that has less than 4 count frequency, which would be SYD and SEL for instance. What would be the way to do so without manually dropping them city by city?

Response:
Here you go with filter
```
df.groupby('city').filter(lambda x : len(x)>3)
Out[1743]:
  city
0  NYC
1  NYC
2  NYC
3  NYC
```
Solution two transform
```
sub_df = df[df.groupby('city').city.transform('count')>3].copy()
# add copy for future warning when you need to modify the sub df
```


Question: Pandas - find first non-null value in column If I have a series that has either NULL or some non-null value. How can I find the 1st row where the value is not NULL so I can report back the datatype to the user. If the value is non-null all values are the same datatype in that series.

Response:
You can use first_valid_index with select by loc:
```
s = pd.Series([np.nan,2,np.nan])
print (s)
0    NaN
1    2.0
2    NaN
dtype: float64

print (s.first_valid_index())
1

print (s.loc[s.first_valid_index()])
2.0

# If your Series contains ALL NaNs, you'll need to check as follows:

s = pd.Series([np.nan, np.nan, np.nan])
idx = s.first_valid_index()  # Will return None
first_valid_value = s.loc[idx] if idx is not None else None
print(first_valid_value)
None
```


Question:
Edit the width of bars using pd.DataFrame.plot() I am making a stacked bar plot using:
```
DataFrame.plot(kind='bar',stacked=True)
```
I want to control width of bars so that the bars are connected to each other like a histogram. I've looked through the documentation but to no avail - any suggestions? Is it possible to do it this way?

Response:
For anyone coming across this question: Since pandas 0.14, plotting with bars has a 'width' command: https://github.com/pydata/pandas/pull/6644 The example above can now be solved simply by using
```
df.plot(kind='bar', stacked=True, width=1)
```
See pandas.DataFrame.plot.bar or pandas.DataFrame.plot with kind='bar'. When changing the width of the bars, it might also be appropriate to change the figure size by specifying the figsize= parameter.


Question:
What is the most efficient way to loop through dataframes with pandas? I want to perform my own complex operations on financial data in dataframes in a sequential manner. For example I am using the following MSFT CSV file taken from Yahoo Finance:
```
Date,Open,High,Low,Close,Volume,Adj Close
2011-10-19,27.37,27.47,27.01,27.13,42880000,27.13
2011-10-18,26.94,27.40,26.80,27.31,52487900,27.31
2011-10-17,27.11,27.42,26.85,26.98,39433400,26.98
2011-10-14,27.31,27.50,27.02,27.27,50947700,27.27

....
```
I then do the following:
```
#!/usr/bin/env python
from pandas import *

df = read_csv('table.csv')

for i, row in enumerate(df.values):
    date = df.index[i]
    open, high, low, close, adjclose = row
    #now perform analysis on open/close based on date, etc..
```
Is that the most efficient way? Given the focus on speed in pandas, I would assume there must be some special function to iterate through the values in a manner that one also retrieves the index (possibly through a generator to be memory efficient)? df.iteritems unfortunately only iterates column by column.

Response:
The newest versions of pandas now include a built-in function for iterating over rows.
```
for index, row in df.iterrows():

    # do some logic here
```
Or, if you want it faster use itertuples() But, unutbu's suggestion to use numpy functions to avoid iterating over rows will produce the fastest code.


Question:
Pandas rolling gives NaN I'm looking at the tutorials on window functions, but I don't quite understand why the following code produces NaNs. If I understand correctly, the code creates a rolling window of size 2. Why do the first, fourth, and fifth rows have NaN? At first, I thought it's because adding NaN with another number would produce NaN, but then I'm not sure why the second row wouldn't be NaN.
```
dft = pd.DataFrame({'B': [0, 1, 2, np.nan, 4]},
                   index=pd.date_range('20130101 09:00:00', periods=5, freq='s'))

```python
dft.rolling(2).sum()
#Output
#B
#2013-01-01 09:00:00  NaN
#2013-01-01 09:00:01  1.0
#2013-01-01 09:00:02  3.0
#2013-01-01 09:00:03  NaN
#2013-01-01 09:00:04  NaN
#```
```

Response:
The first thing to notice is that by default rolling looks for n-1 prior rows of data to aggregate, where n is the window size. If that condition is not met, it will return NaN for the window. This is what's happening at the first row. In the fourth and fifth row, it's because one of the values in the sum is NaN. If you would like to avoid returning NaN, you could pass min_periods=1 to the method which reduces the minimum required number of valid observations in the window to 1 instead of 2:
```

```python
dft.rolling(2, min_periods=1).sum()
#Output
#                       B
#2013-01-01 09:00:00  0.0
#2013-01-01 09:00:01  1.0
#2013-01-01 09:00:02  3.0
#2013-01-01 09:00:03  2.0
#2013-01-01 09:00:04  4.0
#```
```


Question:
is it possible to do fuzzy match merge with python pandas? I have two DataFrames which I want to merge based on a column. However, due to alternate spellings, different number of spaces, absence/presence of diacritical marks, I would like to be able to merge as long as they are similar to one another. Any similarity algorithm will do (soundex, Levenshtein, difflib's). Say one DataFrame has the following data:
```
df1 = DataFrame([[1],[2],[3],[4],[5]], index=['one','two','three','four','five'], columns=['number'])

       number
one         1
two         2
three       3
four        4
five        5

df2 = DataFrame([['a'],['b'],['c'],['d'],['e']], index=['one','too','three','fours','five'], columns=['letter'])

      letter
one        a
too        b
three      c
fours      d
five       e
```
Then I want to get the resulting DataFrame
```
number letter
one         1      a
two         2      b
three       3      c
four        4      d
five        5      e
```

Response:
Similar to @locojay suggestion, you can apply difflib's get_close_matches to df2's index and then apply a join:
```

```python
import difflib
```

```python
difflib.get_close_matches
#Output
#<function difflib.get_close_matches>
```

```python
df2.index = df2.index.map(lambda x: difflib.get_close_matches(x, df1.index)[0])
```

```python
df2
#Output
#letter
#one        a
#two        b
#three      c
#four       d
#five       e
```

```python
df1.join(df2)
#Output
#number letter
#one         1      a
#two         2      b
#three       3      c
#four        4      d
#five        5      e
#```
#. If these were columns, in the same vein you could apply to the column then merge:
#```
#df1 = DataFrame([[1,'one'],[2,'two'],[3,'three'],[4,'four'],[5,'five']], columns=['number', 'name'])
#df2 = DataFrame([['a','one'],['b','too'],['c','three'],['d','fours'],['e','five']], columns=['letter', 'name'])
#
#df2['name'] = df2['name'].apply(lambda x: difflib.get_close_matches(x, df1['name'])[0])
#df1.merge(df2)
#```
```


Question:
How to assign a name to the size() column? I am using .size() on a groupby result in order to count how many items are in each group. I would like the result to be saved to a new column name without manually editing the column names array, how can it be done? This is what I have tried:
```
grpd = df.groupby(['A','B'])
grpd['size'] = grpd.size()
grpd
```
and the error I got: TypeError: 'DataFrameGroupBy' object does not support item assignment (on the second line)

Response:
The .size() built-in method of DataFrameGroupBy objects actually returns a Series object with the group sizes and not a DataFrame. If you want a DataFrame whose column is the group sizes, indexed by the groups, with a custom name, you can use the .to_frame() method and use the desired column name as its argument.
```
grpd = df.groupby(['A','B']).size().to_frame('size')
```
If you wanted the groups to be columns again you could add a .reset_index() at the end.


Question:
Pandas get the most frequent values of a column i have this dataframe:
```
0 name data
1 alex asd
2 helen sdd
3 alex dss
4 helen sdsd
5 john sdadd
```
so i am trying to get the most frequent value or values(in this case its values) so what i do is:
```
dataframe['name'].value_counts().idxmax()
```
but it returns only the value: Alex even if it Helen appears two times as well.

Response:
By using mode
```
df.name.mode()
Out[712]:
0     alex
1    helen
dtype: object
```


Question:
How to loop over grouped Pandas dataframe? DataFrame:
```none
c_os_family_ss c_os_major_is l_customer_id_i
0      Windows 7                         90418
1      Windows 7                         90418
2      Windows 7                         90418
```
Code:
```py
for name, group in df.groupby('l_customer_id_i').agg(lambda x: ','.join(x)):
    print name
    print group
```
I'm trying to just loop over the aggregated data, but I get the error:
```none
ValueError: too many values to unpack
```
I wish to loop over every group. How do I do it?

Response:
df.groupby('l_customer_id_i').agg(lambda x: ','.join(x)) does already return a dataframe, so you cannot loop over the groups anymore. In general: df.groupby(...) returns a GroupBy object (a DataFrameGroupBy or SeriesGroupBy), and with this, you can iterate through the groups (as explained in the docs here). You can do something like:
```
grouped = df.groupby('A')

for name, group in grouped:
    ...
```
When you apply a function on the groupby, in your example df.groupby(...).agg(...) (but this can also be transform, apply, mean, ...), you combine the result of applying the function to the different groups together in one dataframe (the apply and combine step of the 'split-apply-combine' paradigm of groupby). So the result of this will always be again a DataFrame (or a Series depending on the applied function).


Question:
How to print pandas DataFrame without index I want to print the whole dataframe, but I don't want to print the index Besides, one column is datetime type, I just want to print time, not date. The dataframe looks like:
```
User ID           Enter Time   Activity Number
0      123  2014-07-08 00:09:00              1411
1      123  2014-07-08 00:18:00               893
2      123  2014-07-08 00:49:00              1041
```
I want it print as
```
User ID   Enter Time   Activity Number
123         00:09:00              1411
123         00:18:00               893
123         00:49:00              1041
```

Response:
```py
print(df.to_string(index=False))
```


Question:
Pandas: sum up multiple columns into one column without last column If I have a dataframe similar to this one
```
Apples   Bananas   Grapes   Kiwis
2        3         nan      1
1        3         7        nan
nan      nan       2        3
```
I would like to add a column like this
```
Apples   Bananas   Grapes   Kiwis   Fruit Total
2        3         nan      1        6
1        3         7        nan      11
nan      nan       2        3        5
```
I guess you could use df['Apples'] + df['Bananas'] and so on, but my actual dataframe is much larger than this. I was hoping a formula like df['Fruit Total']=df[-4:-1].sum could do the trick in one line of code. That didn't work however. Is there any way to do it without explicitly summing up all columns?

Response:
You can first select by iloc and then sum:
```
df['Fruit Total']= df.iloc[:, -4:-1].sum(axis=1)
print (df)
   Apples  Bananas  Grapes  Kiwis  Fruit Total
0     2.0      3.0     NaN    1.0          5.0
1     1.0      3.0     7.0    NaN         11.0
2     NaN      NaN     2.0    3.0          2.0
```
For sum all columns use:
```
df['Fruit Total']= df.sum(axis=1)
```


Question:
When using a pandas dataframe, how do I add column if does not exist? I'm new to using pandas and am writing a script where I read in a dataframe and then do some computation on some of the columns. Sometimes I will have the column called "Met":
```py
df = pd.read_csv(File,
  sep='\t',
  compression='gzip',
  header=0,
  names=["Chrom", "Site", "coverage", "Met"]
)
```
Other times I will have:
```py
df = pd.read_csv(File,
  sep='\t',
  compression='gzip',
  header=0,
  names=["Chrom", "Site", "coverage", "freqC"]
)
```
I need to do some computation with the "Met" column so if it isn't present I will need to calculate it using:
```py
df['Met'] = df['freqC'] * df['coverage']
```
is there a way to check if the "Met" column is present in the dataframe, and if not add it?

Response:
You check it like this:
```
if 'Met' not in df:
    df['Met'] = df['freqC'] * df['coverage']
```


Question:
How do I append one pandas DataFrame to another? I have a problem with appending of dataframe. I try to execute this code
```
df_all = pd.read_csv('data.csv', error_bad_lines=False, chunksize=1000000)
urls = pd.read_excel('url_june.xlsx')
substr = urls.url.values.tolist()
df_res = pd.DataFrame()
for df in df_all:
    for i in substr:
        res = df[df['url'].str.contains(i)]
        df_res.append(res)
```
And when I try to save df_res I get empty dataframe. df_all looks like
```
ID,"url","used_at","active_seconds"
b20f9412f914ad83b6611d69dbe3b2b4,"mobiguru.ru/phones/apple/comp/32gb/apple_iphone_5s.html",2015-10-01 00:00:25,1
b20f9412f914ad83b6611d69dbe3b2b4,"mobiguru.ru/phones/apple/comp/32gb/apple_iphone_5s.html",2015-10-01 00:00:31,30
f85ce4b2f8787d48edc8612b2ccaca83,"4pda.ru/forum/index.php?showtopic=634566&view=getnewpost",2015-10-01 00:01:49,2
d3b0ef7d85dbb4dbb75e8a5950bad225,"shop.mts.ru/smartfony/mts/smartfon-smart-sprint-4g-sim-lock-white.html?utm_source=admitad&utm_medium=cpa&utm_content=300&utm_campaign=gde_cpa&uid=3",2015-10-01 00:03:19,34
078d388438ebf1d4142808f58fb66c87,"market.yandex.ru/product/12675734/spec?hid=91491&track=char",2015-10-01 00:03:48,2
d3b0ef7d85dbb4dbb75e8a5950bad225,"avito.ru/yoshkar-ola/telefony/mts",2015-10-01 00:04:21,4
d3b0ef7d85dbb4dbb75e8a5950bad225,"shoppingcart.aliexpress.com/order/confirm_order",2015-10-01 00:04:25,1
d3b0ef7d85dbb4dbb75e8a5950bad225,"shoppingcart.aliexpress.com/order/confirm_order",2015-10-01 00:04:26,9
```
and urls looks like
```
url
shoppingcart.aliexpress.com/order/confirm_order
ozon.ru/?context=order_done&number=
lk.wildberries.ru/basket/orderconfirmed
lamoda.ru/checkout/onepage/success/quick
mvideo.ru/confirmation?_requestid=
eldorado.ru/personal/order.php?step=confirm
```
When I print res in a loop it doesn't empty. But when I try print in a loop df_res after append, it return empty dataframe. I can't find my error. How can I fix it?

Response:
If you look at the documentation for pd.DataFrame.append Append rows of other to the end of this frame, returning a new object. Columns not in this frame are added as new columns. (emphasis mine). Try
```
df_res = df_res.append(res)
```
Incidentally, note that pandas isn't that efficient for creating a DataFrame by successive concatenations. You might try this, instead:
```
all_res = []
for df in df_all:
    for i in substr:
        res = df[df['url'].str.contains(i)]
        all_res.append(res)

df_res = pd.concat(all_res)
```
This first creates a list of all the parts, then creates a DataFrame from all of them once at the end.


Question:
Pandas create new column with count from groupby I have a df that looks like the following:
```
id        item        color
01        truck       red
02        truck       red
03        car         black
04        truck       blue
05        car         black
```
I am trying to create a df that looks like this:
```
item      color       count
truck     red          2
truck     blue         1
car       black        2
```
I have tried
```
df["count"] = df.groupby("item")["color"].transform('count')
```
But it is not quite what I am searching for. Any guidance is appreciated

Response:
That's not a new column, that's a new DataFrame:
```

```python
df.groupby(["item", "color"]).count()
#Output
#id
#item  color
#car   black   2
#truck blue    1
#      red     2
#```
#To get the result you want is to use reset_index:
#```
```

```python
df.groupby(["item", "color"])["id"].count().reset_index(name="count")
#Output
#item  color  count
#0    car  black      2
#1  truck   blue      1
#2  truck    red      2
#```
#To get a "new column" you could use transform:
#```
```

```python
df.groupby(["item", "color"])["id"].transform("count")
#Output
#0    2
#1    2
#2    2
#3    1
#4    2
#dtype: int64
#```
#I recommend reading the split-apply-combine section of the docs.
```


Question:
Nested Json to pandas DataFrame with specific format I need to format the contents of a Json file in a certain format in a pandas DataFrame so that I can run pandassql to transform the data and run it through a scoring model. file = C:\scoring_model\json.js (contents of 'file' are below)
```
{
"response":{
  "version":"1.1",
  "token":"dsfgf",
   "body":{
     "customer":{
         "customer_id":"1234567",
         "verified":"true"
       },
     "contact":{
         "email":"
[email protected]
",
         "mobile_number":"0123456789"
      },
     "personal":{
         "gender": "m",
         "title":"Dr.",
         "last_name":"Muster",
         "first_name":"Max",
         "family_status":"single",
         "dob":"1985-12-23",
     }
   }
 }
```
I need the dataframe to look like this (obviously all values on same row, tried to format it best as possible for this question):
```
version | token | customer_id | verified | email      | mobile_number | gender |
1.1     | dsfgf | 1234567     | true     |
[email protected]
 | 0123456789    | m      |

title | last_name | first_name |family_status | dob
Dr.   | Muster    | Max        | single       | 23.12.1985
```
I have looked at all the other questions on this topic, have tried various ways to load Json file into pandas
```
with open(r'C:\scoring_model\json.js', 'r') as f:
    c = pd.read_json(f.read())

with open(r'C:\scoring_model\json.js', 'r') as f:
    c = f.readlines()
```
tried pd.Panel() in this solution Python Pandas: How to split a sorted dictionary in a column of a dataframe with dataframe results from [yo = f.readlines()]. I thought about trying to split contents of each cell based on ("") and find a way to put the split contents into different columns but no luck so far.

Response:
If you load in the entire json as a dict (or list) e.g. using json.load, you can use json_normalize:
```

```python
d = {"response": {"body": {"contact": {"email": "
[email protected]
", "mobile_number": "0123456789"}, "personal": {"last_name": "Muster", "gender": "m", "first_name": "Max", "dob": "1985-12-23", "family_status": "single", "title": "Dr."}, "customer": {"verified": "true", "customer_id": "1234567"}}, "token": "dsfgf", "version": "1.1"}}
```

```python
df = pd.json_normalize(d)
```

```python
df.columns = df.columns.map(lambda x: x.split(".")[-1])
```

```python
df
#Output
#email mobile_number customer_id verified         dob family_status first_name gender last_name title  token version
#0
#[email protected]
#    0123456789     1234567     true  1985-12-23        single        Max      m    Muster   Dr.  dsfgf     1.1
#```
```


Question: How to reversibly store and load a Pandas dataframe to/from disk Right now I'm importing a fairly large CSV as a dataframe every time I run the script. Is there a good solution for keeping that dataframe constantly available in between runs so I don't have to spend all that time waiting for the script to run?

Response:
The easiest way is to pickle it using to_pickle:
```
df.to_pickle(file_name)  # where to save it, usually as a .pkl
```
Then you can load it back using:
```
df = pd.read_pickle(file_name)
```
Note: before 0.11.1 save and load were the only way to do this (they are now deprecated in favor of to_pickle and read_pickle respectively). Another popular choice is to use HDF5 (pytables) which offers very fast access times for large datasets:
```
import pandas as pd
store = pd.HDFStore('store.h5')

store['df'] = df  # save it
store['df']  # load it
```
More advanced strategies are discussed in the cookbook. Since 0.13 there's also msgpack which may be be better for interoperability, as a faster alternative to JSON, or if you have python object/text-heavy data (see this question).


Question:
Pandas Timedelta in Days I have a dataframe in pandas called 'munged_data' with two columns 'entry_date' and 'dob' which i have converted to Timestamps using pd.to_timestamp.I am trying to figure out how to calculate ages of people based on the time difference between 'entry_date' and 'dob' and to do this i need to get the difference in days between the two columns ( so that i can then do somehting like round(days/365.25). I do not seem to be able to find a way to do this using a vectorized operation. When I do munged_data.entry_date-munged_data.dob i get the following :
```
internal_quote_id
2                    15685977 days, 23:54:30.457856
3                    11651985 days, 23:49:15.359744
4                     9491988 days, 23:39:55.621376
7                     11907004 days, 0:10:30.196224
9                    15282164 days, 23:30:30.196224
15                  15282227 days, 23:50:40.261632
```
However i do not seem to be able to extract the days as an integer so that i can continue with my calculation. Any help appreciated.

Response:
Using the Pandas type Timedelta available since v0.15.0 you also can do:
```

```python
import pandas as pd
```

```python
df = pd.DataFrame([ pd.Timestamp('20150111'),
                           pd.Timestamp('20150301') ], columns=['date'])
```

```python
df['today'] = pd.Timestamp('20150315')
```

```python
df
#Output
#date      today
#0 2015-01-11 2015-03-15
#1 2015-03-01 2015-03-15
```

```python
(df['today'] - df['date']).dt.days
#Output
#0    63
#1    14
#dtype: int64
#```
```


Question:
Pandas DataFrame concat vs append I have a list of 4 pandas dataframes containing a day of tick data that I want to merge into a single data frame. I cannot understand the behavior of concat on my timestamps. See details below:
```
data

[<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 35228 entries, 2013-03-28 00:00:07.089000+02:00 to 2013-03-28 18:59:20.357000+02:00
Data columns:
Price       4040  non-null values
Volume      4040  non-null values
BidQty      35228  non-null values
BidPrice    35228  non-null values
AskPrice    35228  non-null values
AskQty      35228  non-null values
dtypes: float64(6),
<class 'pandas.core.frame.DataFrame'>

DatetimeIndex: 33088 entries, 2013-04-01 00:03:17.047000+02:00 to 2013-04-01 18:59:58.175000+02:00
Data columns:
Price       3969  non-null values
Volume      3969  non-null values
BidQty      33088  non-null values
BidPrice    33088  non-null values
AskPrice    33088  non-null values
AskQty      33088  non-null values
dtypes: float64(6),
<class 'pandas.core.frame.DataFrame'>

DatetimeIndex: 50740 entries, 2013-04-02 00:03:27.470000+02:00 to 2013-04-02 18:59:58.172000+02:00
Data columns:
Price       7326  non-null values
Volume      7326  non-null values
BidQty      50740  non-null values
BidPrice    50740  non-null values
AskPrice    50740  non-null values
AskQty      50740  non-null values
dtypes: float64(6),
<class 'pandas.core.frame.DataFrame'>

DatetimeIndex: 60799 entries, 2013-04-03 00:03:06.994000+02:00 to 2013-04-03 18:59:58.180000+02:00
Data columns:
Price       8258  non-null values
Volume      8258  non-null values
BidQty      60799  non-null values
BidPrice    60799  non-null values
AskPrice    60799  non-null values
AskQty      60799  non-null values
dtypes: float64(6)]
```
Using append I get:
```
pd.DataFrame().append(data)

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 179855 entries, 2013-03-28 00:00:07.089000+02:00 to 2013-04-03 18:59:58.180000+02:00
Data columns:
AskPrice    179855  non-null values
AskQty      179855  non-null values
BidPrice    179855  non-null values
BidQty      179855  non-null values
Price       23593  non-null values
Volume      23593  non-null values
dtypes: float64(6)
```
Using concat I get:
```
pd.concat(data)

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 179855 entries, 2013-03-27 22:00:07.089000+02:00 to 2013-04-03 16:59:58.180000+02:00
Data columns:
Price       23593  non-null values
Volume      23593  non-null values
BidQty      179855  non-null values
BidPrice    179855  non-null values
AskPrice    179855  non-null values
AskQty      179855  non-null values
dtypes: float64(6)
```
Notice how the index changes when using concat. Why is that happening and how would I go about using concat to reproduce the results obtained using append? (Since concat seems so much faster; 24.6 ms per loop vs 3.02 s per loop)

Response: Pandas concat vs append vs join vs merge Concat gives the flexibility to join based on the axis( all rows or all columns) Append is the specific case(axis=0, join='outer') of concat (being deprecated use concat) Join is based on the indexes (set by set_index) on how variable =['left','right','inner','couter'] Merge is based on any particular column each of the two dataframes, this columns are variables on like 'left_on', 'right_on', 'on'


Question:
Pandas converting row with unix timestamp (in milliseconds) to datetime I need to process a huge amount of CSV files where the time stamp is always a string representing the unix timestamp in milliseconds. I could not find a method yet to modify these columns efficiently. This is what I came up with, however this of course duplicates only the column and I have to somehow put it back to the original dataset. I'm sure it can be done when creating the DataFrame?
```
import sys
if sys.version_info[0] < 3:
    from StringIO import StringIO
else:
    from io import StringIO
import pandas as pd

data = 'RUN,UNIXTIME,VALUE\n1,1447160702320,10\n2,1447160702364,20\n3,1447160722364,42'

df = pd.read_csv(StringIO(data))

convert = lambda x: datetime.datetime.fromtimestamp(x / 1e3)
converted_df = df['UNIXTIME'].apply(convert)
```
This will pick the column 'UNIXTIME' and change it from
```
0    1447160702320
1    1447160702364
2    1447160722364
Name: UNIXTIME, dtype: int64
```
into this
```
0   2015-11-10 14:05:02.320
1   2015-11-10 14:05:02.364
2   2015-11-10 14:05:22.364
Name: UNIXTIME, dtype: datetime64[ns]
```
However, I would like to use something like pd.apply() to get the whole dataset returned with the converted column or as I already wrote, simply create datetimes when generating the DataFrame from CSV.

Response:
You can do this as a post processing step using to_datetime and passing arg unit='ms':
```

```python
df['UNIXTIME'] = pd.to_datetime(df['UNIXTIME'], unit='ms')
df
#Output
#RUN                UNIXTIME  VALUE
#0    1 2015-11-10 13:05:02.320     10
#1    2 2015-11-10 13:05:02.364     20
#2    3 2015-11-10 13:05:22.364     42
#```
```


Question:
Get total number of hours from a Pandas Timedelta? How can I get the total number of hours in a Pandas timedelta? For example:
```

```python
td = pd.Timedelta('1 days 2 hours')
```

```python
td.get_total_hours()
#Output
#26
#```
#Note: as per the documentation, the .hours attribute will return the hours component:
#```
```

```python
td.hours
#Output
#2
#```
```

Response:
Just find out how many timedeltas of 1 hour fit into it:
```
import numpy as np

>> td / np.timedelta64(1, 'h')
26.0
```


Question:
Is there a way to copy only the structure (not the data) of a Pandas DataFrame? I received a DataFrame from somewhere and want to create another DataFrame with the same number and names of columns and rows (indexes). For example, suppose that the original data frame was created as
```
import pandas as pd
df1 = pd.DataFrame([[11,12],[21,22]], columns=['c1','c2'], index=['i1','i2'])
```
I copied the structure by explicitly defining the columns and names:
```
df2 = pd.DataFrame(columns=df1.columns, index=df1.index)
```
I don't want to copy the data, otherwise I could just write df2 = df1.copy(). In other words, after df2 being created it must contain only NaN elements:
```

```python
df1
#Output
#c1  c2
#i1  11  12
#i2  21  22
```

```python
df2
#Output
#c1   c2
#i1  NaN  NaN
#i2  NaN  NaN
#```
#Is there a more idiomatic way of doing it?
```

Response:
That's a job for reindex_like. Start with the original:
```
df1 = pd.DataFrame([[11, 12], [21, 22]], columns=['c1', 'c2'], index=['i1', 'i2'])
```
Construct an empty DataFrame and reindex it like df1:
```
pd.DataFrame().reindex_like(df1)
Out:
    c1  c2
i1 NaN NaN
i2 NaN NaN
```


Question:
How to create a large pandas dataframe from an sql query without running out of memory? I have trouble querying a table of > 5 million records from MS SQL Server database. I want to select all of the records, but my code seems to fail when selecting to much data into memory. This works:
```
import pandas.io.sql as psql
sql = "SELECT TOP 1000000 * FROM MyTable"
data = psql.read_frame(sql, cnxn)
```
...but this does not work:
```
sql = "SELECT TOP 2000000 * FROM MyTable"
data = psql.read_frame(sql, cnxn)
```
It returns this error:
```
File "inference.pyx", line 931, in pandas.lib.to_object_array_tuples
(pandas\lib.c:42733) Memory Error
```
I have read here that a similar problem exists when creating a dataframe from a csv file, and that the work-around is to use the 'iterator' and 'chunksize' parameters like this:
```
read_csv('exp4326.csv', iterator=True, chunksize=1000)
```
Is there a similar solution for querying from an SQL database? If not, what is the preferred work-around? Should I use some other methods to read the records in chunks? I read a bit of discussion here about working with large datasets in pandas, but it seems like a lot of work to execute a SELECT * query. Surely there is a simpler approach.

Response:
As mentioned in a comment, starting from pandas 0.15, you have a chunksize option in read_sql to read and process the query chunk by chunk:
```
sql = "SELECT * FROM My_Table"
for chunk in pd.read_sql_query(sql , engine, chunksize=5):
    print(chunk)
```
Reference: http://pandas.pydata.org/pandas-docs/version/0.15.2/io.html#querying


Question:
Pandas: assign an index to each group identified by groupby When using groupby(), how can I create a DataFrame with a new column containing an index of the group number, similar to dplyr::group_indices in R. For example, if I have
```

```python
df=pd.DataFrame({'a':[1,1,1,2,2,2],'b':[1,1,2,1,1,2]})
```

```python
df
#Output
#   a  b
#0  1  1
#1  1  1
#2  1  2
#3  2  1
#4  2  1
#5  2  2
#```
#How can I get a DataFrame like
#```
#a  b  idx
#0  1  1  1
#1  1  1  1
#2  1  2  2
#3  2  1  3
#4  2  1  3
#5  2  2  4
#```
#(the order of the idx indexes doesn't matter)
```

Response:
Here is the solution using ngroup (available as of pandas 0.20.2) from a comment above by Constantino.
```
import pandas as pd
df = pd.DataFrame({'a':[1,1,1,2,2,2],'b':[1,1,2,1,1,2]})
df['idx'] = df.groupby(['a', 'b']).ngroup()
df
```
```
a  b  idx
0  1  1    0
1  1  1    0
2  1  2    1
3  2  1    2
4  2  1    2
5  2  2    3
```


Question:
Pandas: Subtracting two date columns and the result being an integer I have two columns in a Pandas data frame that are dates. I am looking to subtract one column from another and the result being the difference in numbers of days as an integer. A peek at the data:
```
df_test.head(10)
Out[20]:
  First_Date Second Date
0 2016-02-09  2015-11-19
1 2016-01-06  2015-11-30
2        NaT  2015-12-04
3 2016-01-06  2015-12-08
4        NaT  2015-12-09
5 2016-01-07  2015-12-11
6        NaT  2015-12-12
7        NaT  2015-12-14
8 2016-01-06  2015-12-14
9        NaT  2015-12-15
```
I have created a new column successfully with the difference:
```
df_test['Difference'] = df_test['First_Date'].sub(df_test['Second Date'], axis=0)
df_test.head()
Out[22]:
  First_Date Second Date  Difference
0 2016-02-09  2015-11-19     82 days
1 2016-01-06  2015-11-30     37 days
2        NaT  2015-12-04         NaT
3 2016-01-06  2015-12-08     29 days
4        NaT  2015-12-09         NaT
```
However I am unable to get a numeric version of the result:
```
df_test['Difference'] = df_test[['Difference']].apply(pd.to_numeric)

df_test.head()
Out[25]:
  First_Date Second Date    Difference
0 2016-02-09  2015-11-19  7.084800e+15
1 2016-01-06  2015-11-30  3.196800e+15
2        NaT  2015-12-04           NaN
3 2016-01-06  2015-12-08  2.505600e+15
4        NaT  2015-12-09           NaN
```

Response:
How about:
```
df_test['Difference'] = (df_test['First_Date'] - df_test['Second Date']).dt.days
```
This will return difference as int if there are no missing values(NaT) and float if there is. Pandas have a rich documentation on Time series / date functionality and Time deltas


Question:
return column names from pyodbc execute() statement
```
from pandas import DataFrame
import pyodbc

cnxn = pyodbc.connect(databasez)
cursor.execute("""SELECT ID, NAME AS Nickname, ADDRESS AS Residence FROM tablez""")
DF = DataFrame(cursor.fetchall())
```
This is fine to populate my pandas DataFrame. But how do I get
```
DF.columns = ['ID', 'Nickname', 'Residence']
```
straight from cursor? Is that information stored in cursor at all?

Response: You can get the columns from the cursor description: columns = [column[0] for column in cursor.description]


Question:
Rolling Mean on pandas on a specific column I have a data frame like this which is imported from a CSV.
```
stock  pop
Date
2016-01-04  325.316   82
2016-01-11  320.036   83
2016-01-18  299.169   79
2016-01-25  296.579   84
2016-02-01  295.334   82
2016-02-08  309.777   81
2016-02-15  317.397   75
2016-02-22  328.005   80
2016-02-29  315.504   81
2016-03-07  328.802   81
2016-03-14  339.559   86
2016-03-21  352.160   82
2016-03-28  348.773   84
2016-04-04  346.482   83
2016-04-11  346.980   80
2016-04-18  357.140   75
2016-04-25  357.439   77
2016-05-02  356.443   78
2016-05-09  365.158   78
2016-05-16  352.160   72
2016-05-23  344.540   74
2016-05-30  354.998   81
2016-06-06  347.428   77
2016-06-13  341.053   78
2016-06-20  363.515   80
2016-06-27  349.669   80
2016-07-04  371.583   82
2016-07-11  358.335   81
2016-07-18  362.021   79
2016-07-25  368.844   77
...             ...  ...
```
I wanted to add a new column MA which calculates Rolling mean for the column pop. I tried the following
```
df['MA']=data.rolling(5,on='pop').mean()
```
I get an error
```
ValueError: Wrong number of items passed 2, placement implies 1
```
So I thought let me try if it just works without adding a column. I used
```
data.rolling(5,on='pop').mean()
```
I got the output
```
stock  pop
Date
2016-01-04       NaN   82
2016-01-11       NaN   83
2016-01-18       NaN   79
2016-01-25       NaN   84
2016-02-01  307.2868   82
2016-02-08  304.1790   81
2016-02-15  303.6512   75
2016-02-22  309.4184   80
2016-02-29  313.2034   81
2016-03-07  319.8970   81
2016-03-14  325.8534   86
2016-03-21  332.8060   82
2016-03-28  336.9596   84
2016-04-04  343.1552   83
2016-04-11  346.7908   80
2016-04-18  350.3070   75
2016-04-25  351.3628   77
2016-05-02  352.8968   78
2016-05-09  356.6320   78
2016-05-16  357.6680   72
2016-05-23  355.1480   74
2016-05-30  354.6598   81
2016-06-06  352.8568   77
2016-06-13  348.0358   78
2016-06-20  350.3068   80
2016-06-27  351.3326   80
2016-07-04  354.6496   82
2016-07-11  356.8310   81
2016-07-18  361.0246   79
2016-07-25  362.0904   77
...              ...  ...
```
I can't seem to apply Rolling mean on the column pop. What am I doing wrong?

Response:
To assign a column, you can create a rolling object based on your Series:
```
df['new_col'] = data['column'].rolling(5).mean()
```
The answer posted by ac2001 is not the most performant way of doing this. He is calculating a rolling mean on every column in the dataframe, then he is assigning the "ma" column using the "pop" column. The first method of the following is much more efficient:
```
%timeit df['ma'] = data['pop'].rolling(5).mean()
%timeit df['ma_2'] = data.rolling(5).mean()['pop']

1000 loops, best of 3: 497 µs per loop
100 loops, best of 3: 2.6 ms per loop
```
I would not recommend using the second method unless you need to store computed rolling means on all other columns.


Question:
Filter string data based on its string length I like to filter out data whose string length is not equal to 10. If I try to filter out any row whose column A's or B's string length is not equal to 10, I tried this.
```
df=pd.read_csv('filex.csv')
df.A=df.A.apply(lambda x: x if len(x)== 10 else np.nan)
df.B=df.B.apply(lambda x: x if len(x)== 10 else np.nan)
df=df.dropna(subset=['A','B'], how='any')
```
This works slow, but is working. However, it sometimes produce error when the data in A is not a string but a number (interpreted as a number when read_csv read the input file):
```
File "<stdin>", line 1, in <lambda>
TypeError: object of type 'float' has no len()
```
I believe there should be more efficient and elegant code instead of this. Based on the answers and comments below, the simplest solution I found are:
```
df=df[df.A.apply(lambda x: len(str(x))==10]
df=df[df.B.apply(lambda x: len(str(x))==10]
```
or
```
df=df[(df.A.apply(lambda x: len(str(x))==10) & (df.B.apply(lambda x: len(str(x))==10)]
```
or
```
df=df[(df.A.astype(str).str.len()==10) & (df.B.astype(str).str.len()==10)]
```

Response:
```
import pandas as pd

df = pd.read_csv('filex.csv')
df['A'] = df['A'].astype('str')
df['B'] = df['B'].astype('str')
mask = (df['A'].str.len() == 10) & (df['B'].str.len() == 10)
df = df.loc[mask]
print(df)
```
Applied to filex.csv:
```
A,B
123,abc
1234,abcd
1234567890,abcdefghij
```
the code above prints
```
A           B
2  1234567890  abcdefghij
```


Question: How can I create stacked line graph? I would like to be able to produce a stacked line graph (similar to the method used here) with Python (preferably using matplotlib, but another library would be fine too). How can I do this? This similar to the stacked bar graph example on their website, except I'd like the top of bar to be connected with a line segment and the area underneath to be filled. I might be able to approximate this by decreasing the gaps between bars and using lots of bars (but this seems like a hack, and besides I'm not sure if it is possible).

Response:
Newer versions of matplotlib contain the function plt.stackplot(), which allows for several different "out-of-the-box" stacked area plots:
```
import numpy as np
import pylab as plt

X = np.arange(0, 10, 1)
Y = X + 5 * np.random.random((5, X.size))

baseline = ["zero", "sym", "wiggle", "weighted_wiggle"]
for n, v in enumerate(baseline):
    plt.subplot(2 ,2, n + 1)
    plt.stackplot(X, *Y, baseline=v)
    plt.title(v)
    plt.axis('tight')
plt.show()
```


Question:
Querying for NaN and other names in Pandas Say I have a dataframe df with a column value holding some float values and some NaN. How can I get the part of the dataframe where we have NaN using the query syntax? The following, for example, does not work:
```
df.query( '(value < 10) or (value == NaN)' )
```
I get name NaN is not defined (same for df.query('value ==NaN')) Generally speaking, is there any way to use numpy names in query, such as inf, nan, pi, e, etc.?

Response:
In general, you could use @local_variable_name, so something like
```

```python
pi = np.pi; nan = np.nan
```

```python
df = pd.DataFrame({"value": [3,4,9,10,11,np.nan,12]})
```

```python
df.query("(value < 10) and (value > @pi)")
#Output
#   value
#1      4
#2      9
#```
#would work, but nan isn't equal to itself, so value == NaN will always be false. One way to hack around this is to use that fact, and use value != value as an isnan check. We have
#```
```

```python
df.query("(value < 10) or (value == @nan)")
#Output
#   value
#0      3
#1      4
#2      9
#```
#but
#```
```

```python
df.query("(value < 10) or (value != value)")
#Output
#   value
#0      3
#1      4
#2      9
#5    NaN
#```
```


Question:
Get frequency of item occurrences in a column as percentage [duplicate] This question already has answers here: Given a pandas Series that represents frequencies of a value, how can I turn those frequencies into percentages? (4 answers) Pandas percentage of total with groupby (17 answers) Closed 3 years ago. I want to get a percentage of a particular value in a df column. Say I have a df with (col1, col2 , col3, gender) gender column has values of M, F, or Other. I want to get the percentage of M, F, Other values in the df. I have tried this, which gives me the number M, F, Other instances, but I want these as a percentage of the total number of values in the df.
```
df.groupby('gender').size()
```
Can someone help?

Response:
Use value_counts with normalize=True:
```
df['gender'].value_counts(normalize=True) * 100
```
The result is a fraction in range (0, 1]. We multiply by 100 here in order to get the %.


Question:
converting list of header and row lists into pandas DataFrame I am reading contents of a spreadsheet into pandas. DataNitro has a method that returns a rectangular selection of cells as a list of lists. So
```
table = Cell("A1").table
```
gives
```
table = [['Heading1', 'Heading2'], [1 , 2], [3, 4]]

headers = table.pop(0) # gives the headers as list and leaves data
```
I am busy writing code to translate this, but my guess is that it is such a simple use that there must be method to do this. Cant seem to find it in documentation. Any pointers to the method that would simplify this?

Response:
Call the pd.DataFrame constructor directly:
```
df = pd.DataFrame(table, columns=headers)
df

   Heading1  Heading2
0         1         2
1         3         4
```


Question:
Opposite of melt in python pandas I cannot figure out how to do "reverse melt" using Pandas in python. This is my starting data
```
label type  value
0     x    a      1
1     x    b      2
2     x    c      3
3     y    a      4
4     y    b      5
5     y    c      6
6     z    a      7
7     z    b      8
8     z    c      9
```
This is the output I would like to have:
```
label   a   b   c
    x   1   2   3
    y   4   5   6
    z   7   8   9
```
I'm sure there is an easy way to do this, but I don't know how.

Response:
there are a few ways: using .pivot:
```

```python
origin.pivot(index='label', columns='type')['value']
#Output
#type   a  b  c
#label
#x      1  2  3
#y      4  5  6
#z      7  8  9
```

[3 rows x 3 columns]
```
using pivot_table:
```

```python
origin.pivot_table(values='value', index='label', columns='type')
#Output
#       value
#type       a  b  c
#label
#x          1  2  3
#y          4  5  6
#z          7  8  9
```

[3 rows x 3 columns]
```
or .groupby followed by .unstack:
```

```python
origin.groupby(['label', 'type'])['value'].aggregate('mean').unstack()
#Output
#type   a  b  c
#label
#x      1  2  3
#y      4  5  6
#z      7  8  9
```

[3 rows x 3 columns]
```


Question:
pandas groupby, then sort within groups I want to group my dataframe by two columns and then sort the aggregated results within those groups.
```

```python
df
#Output
#count     job source
#0      2   sales      A
#1      4   sales      B
#2      6   sales      C
#3      3   sales      D
#4      7   sales      E
#5      5  market      A
#6      3  market      B
#7      2  market      C
#8      4  market      D
#9      1  market      E
```

```python
df.groupby(['job','source']).agg({'count':sum})
#Output
#count
#job    source
#market A           5
#       B           3
#       C           2
#       D           4
#       E           1
#sales  A           2
#       B           4
#       C           6
#       D           3
#       E           7
#```
#I would now like to sort the 'count' column in descending order within each of the groups, and then take only the top three rows. To get something like:
#```
#count
#job     source
#market  A           5
#        D           4
#        B           3
#sales   E           7
#        C           6
#        B           4
#```
```

Response:
You could also just do it in one go, by doing the sort first and using head to take the first 3 of each group.
```

```python
df.sort_values(['job','count'],ascending=False).groupby('job').head(3)
#Output
#count     job source
#4      7   sales      E
#2      6   sales      C
#1      4   sales      B
#5      5  market      A
#8      4  market      D
#6      3  market      B
#```
```


Question:
How to select all columns except one in pandas? I have a dataframe that look like this:
```none
a         b         c         d
0  0.418762  0.042369  0.869203  0.972314
1  0.991058  0.510228  0.594784  0.534366
2  0.407472  0.259811  0.396664  0.894202
3  0.726168  0.139531  0.324932  0.906575
```
How I can get all columns except b?

Response:
When the columns are not a MultiIndex, df.columns is just an array of column names so you can do:
```
df.loc[:, df.columns != 'b']

          a         c         d
0  0.561196  0.013768  0.772827
1  0.882641  0.615396  0.075381
2  0.368824  0.651378  0.397203
3  0.788730  0.568099  0.869127
```


Question:
Rename Pandas DataFrame Index I've a csv file without header, with a DateTime index. I want to rename the index and column name, but with df.rename() only the column name is renamed. Bug? I'm on version 0.12.0
```

```python
df = pd.read_csv(r'D:\Data\DataTimeSeries_csv//seriesSM.csv', header=None, parse_dates=[[0]], index_col=[0] )
```

```python
df.head()
#Output
#1
#0
#2002-06-18  0.112000
#2002-06-22  0.190333
#2002-06-26  0.134000
#2002-06-30  0.093000
#2002-07-04  0.098667
```

```python
df.rename(index={0:'Date'}, columns={1:'SM'}, inplace=True)
```

```python
df.head()
#Output
#SM
#0
#2002-06-18  0.112000
#2002-06-22  0.190333
#2002-06-26  0.134000
#2002-06-30  0.093000
#2002-07-04  0.098667
#```
```

Response:
The rename method takes a dictionary for the index which applies to index values. You want to rename to index level's name:
```
df.index.names = ['Date']
```
A good way to think about this is that columns and index are the same type of object (Index or MultiIndex), and you can interchange the two via transpose. This is a little bit confusing since the index names have a similar meaning to columns, so here are some more examples:
```

```python
df = pd.DataFrame([[1, 2, 3], [4, 5 ,6]], columns=list('ABC'))
```

```python
df
#Output
#A  B  C
#0  1  2  3
#1  4  5  6
```

```python
df1 = df.set_index('A')
```

```python
df1
#Output
#B  C
#A
#1  2  3
#4  5  6
#```
#You can see the rename on the index, which can change the value 1:
#```
```

```python
df1.rename(index={1: 'a'})
#Output
#B  C
#A
#a  2  3
#4  5  6
```

```python
df1.rename(columns={'B': 'BB'})
#Output
#BB  C
#A
#1   2  3
#4   5  6
#```
#Whilst renaming the level names:
#```
```

```python
df1.index.names = ['index']
        df1.columns.names = ['column']
```
Note: this attribute is just a list, and you could do the renaming as a list comprehension/map.
```
```

```python
df1
#Output
#column  B  C
#index
#1       2  3
#4       5  6
#```
```


Question:
How do I add a title and axis labels to Seaborn Heatmap? I want to add a title to a seaborn heatmap. Using Pandas and iPython Notebook code is below,
```
a1_p = a1.pivot_table( index='Postcode', columns='Property Type', values='Count', aggfunc=np.mean, fill_value=0)

sns.heatmap(a1_p, cmap="YlGnBu")
```
the data is pretty straight forward:
```

```python
a1_p
#Output
#Property Type   Flat    Terraced house  Unknown
#Postcode
#E1  11  0   0
#E14 12  0   0
#E1W 6   0   0
#E2  6   0   0
#```
```

Response:
heatmap is an axes-level function, so you should be able to use just plt.title or ax.set_title:
```
%matplotlib inline
import numpy as np
import os
import seaborn as sns
import matplotlib.pyplot as plt

data = np.random.randn(10,12)

ax = plt.axes()
sns.heatmap(data, ax = ax)

ax.set_title('lalala')
plt.show()
```


Question:
Multi Index Sorting in Pandas I have a dataset with multi-index columns in a pandas df that I would like to sort by values in a specific column. My dataset looks like:
```none
Group1    Group2
    A B C     A B C
1   1 0 3     2 5 7
2   5 6 9     1 0 0
3   7 0 2     0 3 5
```
I want to sort all data and the index by column C in Group 1 in descending order so my results look like:
```none
Group1    Group2
   A B C     A B C
2  5 6 9     1 0 0
1  1 0 3     2 5 7
3  7 0 2     0 3 5
```
Is it possible to do this sort with the structure that my data is in, or should I be swapping Group1 to the index side?

Response:
When sorting by a MultiIndex you need to contain the tuple describing the column inside a list*:
```

```python
df.sort_values([('Group1', 'C')], ascending=False)
#Output
#Group1       Group2
#       A  B  C      A  B  C
#2      5  6  9      1  0  0
#1      1  0  3      2  5  7
#3      7  0  2      0  3  5
#```
#* so as not to confuse pandas into thinking you want to sort first by Group1 then by C. Note: Originally used .sort since deprecated then removed in 0.20, in favor of .sort_values.
```


Question:
Remove first x number of characters from each row in a column of a Python dataframe I have a pandas dataframe with about 1,500 rows and 15 columns. For one specific column, I would like to remove the first 3 characters of each row. As a simple example here is a dataframe:
```py
import pandas as pd

d = {
    'Report Number':['8761234567', '8679876543','8994434555'],
    'Name'         :['George', 'Bill', 'Sally']
     }

d = pd.DataFrame(d)
```
I would like to remove the first three characters from each field in the Report Number column of dataframe d.

Response:
Use vectorised str methods to slice each string entry
```

```python
d['Report Number'] = d['Report Number'].str[3:]
d
#Output
#Name Report Number
#0  George       1234567
#1    Bill       9876543
#2   Sally       4434555
#```
```


Question:
Pandas Dataframe Find Rows Where all Columns Equal I have a dataframe that has characters in it - I want a boolean result by row that tells me if all columns for that row have the same value. For example, I have
```
df = [  a   b   c   d

0  'C'   'C'   'C'   'C'

1  'C'   'C'   'A'   'A'

2  'A'   'A'   'A'   'A' ]
```
and I want the result to be
```
0  True

1  False

2  True
```
I've tried .all but it seems I can only check if all are equal to one letter. The only other way I can think of doing it is by doing a unique on each row and see if that equals 1? Thanks in advance.

Response:
I think the cleanest way is to check all columns against the first column using eq:
```

```python
df
#Output
#a  b  c  d
#0  C  C  C  C
#1  C  C  A  A
#2  A  A  A  A
```

```python
df.iloc[:, 0]
#Output
#0    C
#1    C
#2    A
#Name: a, dtype: object
```

```python
df.eq(df.iloc[:, 0], axis=0)
#Output
#a     b      c      d
#0  True  True   True   True
#1  True  True  False  False
#2  True  True   True   True
#```
#Now you can use all (if they are all equal to the first item, they are all equal):
#```
```

```python
df.eq(df.iloc[:, 0], axis=0).all(1)
#Output
#0     True
#1    False
#2     True
#dtype: bool
#```
```


Question:
Looking for pandas "ungroup by" operation opposite to .groupby in the following string aggregation? Suppose we take a pandas dataframe...
```
name  age  family
0   john    1       1
1  jason   36       1
2   jane   32       1
3   jack   26       2
4  james   30       2
```
Then do a groupby() ...
```
group_df = df.groupby('family')
group_df = group_df.aggregate({'name': name_join, 'age': pd.np.mean})
```
Then do some aggregate/summarize operation (in my example, my function name_join aggregates the names):
```
def name_join(list_names, concat='-'):
    return concat.join(list_names)
```
The grouped summarized output is thus:
```
age             name
family
1        23  john-jason-jane
2        28       jack-james
```
Question: Is there a quick, efficient way to get to the following from the aggregated table?
```
name  age  family
0   john   23       1
1  jason   23       1
2   jane   23       1
3   jack   28       2
4  james   28       2
```
(Note: the age column values are just examples, I don't care for the information I am losing after averaging in this specific example)

Response: The rough equivalent is .reset_index(), but it may not be helpful to think of it as the "opposite" of groupby(). You are splitting a string in to pieces, and maintaining each piece's association with 'family'. This old answer of mine does the job. Just set 'family' as the index column first, refer to the link above, and then reset_index() at the end to get your desired result.


Question: How to read categorical columns with pandas' read_csv? I have tried passing the dtype parameter with read_csv as dtype={n: pandas.Categorical} but this does not work properly (the result is an Object). The manual is unclear. Is it possible to read categorical columns with pd.read_csv?

Response:
In version 0.19.0 you can use parameter dtype='category' in read_csv:
```
data = 'col1,col2,col3\na,b,1\na,b,2\nc,d,3'
df = pd.read_csv(pd.compat.StringIO(data), dtype='category')
print (df)
  col1 col2 col3
0    a    b    1
1    a    b    2
2    c    d    3

print (df.dtypes)
col1    category
col2    category
col3    category
dtype: object
```
If want specify column for category use dtype with dictionary:
```
df = pd.read_csv(pd.compat.StringIO(data), dtype={'col1':'category'})
print (df)
  col1 col2  col3
0    a    b     1
1    a    b     2
2    c    d     3

print (df.dtypes)
col1    category
col2      object
col3       int64
dtype: object
```


Question:
Pandas merge two dataframes with different columns I'm surely missing something simple here. Trying to merge two dataframes in pandas that have mostly the same column names, but the right dataframe has some columns that the left doesn't have, and vice versa.
```
>df_may

  id  quantity  attr_1  attr_2
0  1        20       0       1
1  2        23       1       1
2  3        19       1       1
3  4        19       0       0

>df_jun

  id  quantity  attr_1  attr_3
0  5         8       1       0
1  6        13       0       1
2  7        20       1       1
3  8        25       1       1
```
I've tried joining with an outer join:
```
mayjundf = pd.DataFrame.merge(df_may, df_jun, how="outer")
```
But that yields:
```
Left data columns not unique: Index([....
```
I've also specified a single column to join on (on = "id", e.g.), but that duplicates all columns except id like attr_1_x, attr_1_y, which is not ideal. I've also passed the entire list of columns (there are many) to on:
```
mayjundf = pd.DataFrame.merge(df_may, df_jun, how="outer", on=list(df_may.columns.values))
```
Which yields:
```
ValueError: Buffer has wrong number of dimensions (expected 1, got 2)
```
What am I missing? I'd like to get a df with all rows appended, and attr_1, attr_2, attr_3 populated where possible, NaN where they don't show up. This seems like a pretty typical workflow for data munging, but I'm stuck.

Response:
I think in this case concat is what you want:
```

```python
pd.concat([df,df1], axis=0, ignore_index=True)
#Output
#attr_1  attr_2  attr_3  id  quantity
#0       0       1     NaN   1        20
#1       1       1     NaN   2        23
#2       1       1     NaN   3        19
#3       0       0     NaN   4        19
#4       1     NaN       0   5         8
#5       0     NaN       1   6        13
#6       1     NaN       1   7        20
#7       1     NaN       1   8        25
#```
#by passing axis=0 here you are stacking the df's on top of each other which I believe is what you want then producing NaN value where they are absent from their respective dfs.
```


Question:
Pandas: drop columns with all NaN's I have this DataFrame:
```
0   1   2         3   4       5   6          7
0               #0915-8 NaN NaN       NaN NaN     NaN NaN        NaN
1                   NaN NaN NaN  LIVE WGT NaN  AMOUNT NaN      TOTAL
2               GBW COD NaN NaN     2,280 NaN   $0.60 NaN  $1,368.00
3               POLLOCK NaN NaN     1,611 NaN   $0.01 NaN     $16.11
4                 WHAKE NaN NaN       441 NaN   $0.70 NaN    $308.70
5           GBE HADDOCK NaN NaN     2,788 NaN   $0.01 NaN     $27.88
6           GBW HADDOCK NaN NaN    16,667 NaN   $0.01 NaN    $166.67
7               REDFISH NaN NaN       932 NaN   $0.01 NaN      $9.32
8    GB WINTER FLOUNDER NaN NaN       145 NaN   $0.25 NaN     $36.25
9   GOM WINTER FLOUNDER NaN NaN    25,070 NaN   $0.35 NaN  $8,774.50
10        GB YELLOWTAIL NaN NaN        26 NaN   $1.75 NaN     $45.50
```
I want to drop all NaNs as well as any columns with more than 3 NaNs (either one, or both, should work I think). I tried this code:
```
fish_frame.dropna()
fish_frame.dropna(thresh=len(fish_frame) - 3, axis=1)
```
but it seems not to have any effect on the DataFrame - I see the same results afterward. What is wrong with the code, and how do I fix it?

Response:
From the dropna docstring: Drop the columns where all elements are NaN:
```
df.dropna(axis=1, how='all')

   A    B    D
0  NaN  2.0  0
1  3.0  4.0  1
2  NaN  NaN  5
```


Question:
extracting days from a numpy.timedelta64 value I am using pandas/python and I have two date time series s1 and s2, that have been generated using the 'to_datetime' function on a field of the df containing dates/times. When I subtract s1 from s2 s3 = s2 - s1 I get a series, s3, of type timedelta64[ns]
```
0    385 days, 04:10:36
1     57 days, 22:54:00
2    642 days, 21:15:23
3    615 days, 00:55:44
4    160 days, 22:13:35
5    196 days, 23:06:49
6     23 days, 22:57:17
7      2 days, 22:17:31
8    622 days, 01:29:25
9     79 days, 20:15:14
10    23 days, 22:46:51
11   268 days, 19:23:04
12                  NaT
13                  NaT
14   583 days, 03:40:39
```
How do I look at 1 element of the series: s3[10] I get something like this: numpy.timedelta64(2069211000000000,'ns') How do I extract days from s3 and maybe keep them as integers(not so interested in hours/mins etc.)?

Response:
You can convert it to a timedelta with a day precision. To extract the integer value of days you divide it with a timedelta of one day.
```

```python
x = np.timedelta64(2069211000000000, 'ns')
```

```python
days = x.astype('timedelta64[D]')
```

```python
days / np.timedelta64(1, 'D')
#Output
#23
#```
#Or, as @PhillipCloud suggested, just days.astype(int) since the timedelta is just a 64bit integer that is interpreted in various ways depending on the second parameter you passed in ('D', 'ns', ...). You can find more about it here.
```


Question: How to explode a list inside a Dataframe cell into separate rows I'm looking to turn a pandas cell containing a list into rows for each of those values. So, take this: If I'd like to unpack and stack the values in the nearest_neighbors column so that each value would be a row within each opponent index, how would I best go about this? Are there pandas methods that are meant for operations like this?

Response:
Exploding a list-like column has been simplified significantly in pandas 0.25 with the addition of the explode() method:
```
df = (pd.DataFrame({'name': ['A.J. Price'] * 3,
                    'opponent': ['76ers', 'blazers', 'bobcats'],
                    'nearest_neighbors': [['Zach LaVine', 'Jeremy Lin', 'Nate Robinson', 'Isaia']] * 3})
      .set_index(['name', 'opponent']))

df.explode('nearest_neighbors')
```
Out:
```
nearest_neighbors
name       opponent
A.J. Price 76ers          Zach LaVine
           76ers           Jeremy Lin
           76ers        Nate Robinson
           76ers                Isaia
           blazers        Zach LaVine
           blazers         Jeremy Lin
           blazers      Nate Robinson
           blazers              Isaia
           bobcats        Zach LaVine
           bobcats         Jeremy Lin
           bobcats      Nate Robinson
           bobcats              Isaia
```


Question:
Prepend a level to a pandas MultiIndex I have a DataFrame with a MultiIndex created after some grouping:
```py
import numpy as np
import pandas as pd
from numpy.random import randn

df = pd.DataFrame({'A' : ['a1', 'a1', 'a2', 'a3'],
                   'B' : ['b1', 'b2', 'b3', 'b4'],
                   'Vals' : randn(4)}
                 ).groupby(['A', 'B']).sum()

#            Vals
# A  B
# a1 b1 -1.632460
#    b2  0.596027
# a2 b3 -0.619130
# a3 b4 -0.002009
```
How do I prepend a level to the MultiIndex so that I turn it into something like:
```py
#                       Vals
# FirstLevel A  B
# Foo        a1 b1 -1.632460
#               b2  0.596027
#            a2 b3 -0.619130
#            a3 b4 -0.002009
```

Response:
A nice way to do this in one line using pandas.concat():
```
import pandas as pd

pd.concat([df], keys=['Foo'], names=['Firstlevel'])
```
An even shorter way:
```
pd.concat({'Foo': df}, names=['Firstlevel'])
```
This can be generalized to many data frames, see the docs.