Question:
Replacing values greater than a number in pandas dataframe I have a large dataframe which looks as:
```
df1['A'].ix[1:3]
2017-01-01 02:00:00    [33, 34, 39]
2017-01-01 03:00:00    [3, 43, 9]
```
I want to replace each element greater than 9 with 11. So, the desired output for above example is:
```
df1['A'].ix[1:3]
2017-01-01 02:00:00    [11, 11, 11]
2017-01-01 03:00:00    [3, 11, 9]
```
Edit: My actual dataframe has about 20,000 rows and each row has list of size 2000. Is there a way to use numpy.minimum function for each row? I assume that it will be faster than list comprehension method?

Response:
You can use apply with list comprehension:
```
df1['A'] = df1['A'].apply(lambda x: [y if y <= 9 else 11 for y in x])
print (df1)
                                A
2017-01-01 02:00:00  [11, 11, 11]
2017-01-01 03:00:00    [3, 11, 9]
```
Faster solution is first convert to numpy array and then use numpy.where:
```
a = np.array(df1['A'].values.tolist())
print (a)
[[33 34 39]
 [ 3 43  9]]

df1['A'] = np.where(a > 9, 11, a).tolist()
print (df1)
                                A
2017-01-01 02:00:00  [11, 11, 11]
2017-01-01 03:00:00    [3, 11, 9]
```


Question:
Constructing a co-occurrence matrix in python pandas I know how to do this in R. But, is there any function in pandas that transforms a dataframe to an nxn co-occurrence matrix containing the counts of two aspects co-occurring. For example a matrix df:
```
import pandas as pd

df = pd.DataFrame({'TFD' : ['AA', 'SL', 'BB', 'D0', 'Dk', 'FF'],
                    'Snack' : ['1', '0', '1', '1', '0', '0'],
                    'Trans' : ['1', '1', '1', '0', '0', '1'],
                    'Dop' : ['1', '0', '1', '0', '1', '1']}).set_index('TFD')

print df

```python

#Output
#    Dop Snack Trans
#TFD
#AA    1     1     1
#SL    0     0     1
#BB    1     1     1
#D0    0     1     0
#Dk    1     0     0
#FF    1     0     1
```

[6 rows x 3 columns]
```
would yield:
```
Dop Snack Trans

Dop   0     2     3
Snack 2     0     2
Trans 3     2     0
```
Since the matrix is mirrored on the diagonal I guess there would be a way to optimize code.

Response:
It's a simple linear algebra, you multiply matrix with its transpose (your example contains strings, don't forget to convert them to integer):
```

```python
df_asint = df.astype(int)
```

```python
coocc = df_asint.T.dot(df_asint)
```

```python
coocc
#Output
#       Dop  Snack  Trans
#Dop      4      2      3
#Snack    2      3      2
#Trans    3      2      4
#```
#if, as in R answer, you want to reset diagonal, you can use numpy's fill_diagonal:
#```
```

```python
import numpy as np
```

```python
np.fill_diagonal(coocc.values, 0)
```

```python
coocc
#Output
#       Dop  Snack  Trans
#Dop      0      2      3
#Snack    2      0      2
#Trans    3      2      0
#```
```


Question: How to transform Dask.DataFrame to pd.DataFrame? How can I transform my resulting dask.DataFrame into pandas.DataFrame (let's say I am done with heavy lifting, and just want to apply sklearn to my aggregate result)?

Response:
You can call the .compute() method to transform a dask.dataframe to a pandas dataframe:
```
df = df.compute()
```


Question:
Pandas dataframe: truncate string fields I have a dataframe and would like to truncate each field to up to 20 characters. I've been naively trying the following:
```
df = df.astype(str).apply(lambda x: x[:20])
```
however it has no effect whatsoever. If, however, I wanted to add an 'Y' to each field, this works like a charm:
```
df = df.astype(str).apply(lambda x: x+'Y')
```
What am I doing wrong?

Response:
Simple one liner to trim long string field in Pandas DataFrame:
```
df['short_str'] = df['long_str'].str.slice(0,3)
```


Question:
In pandas, how can I reset index without adding a new column?
```

```python
df = pd.DataFrame([[1, 2, 3, 4], [2, 3, 4, 5], [3, 4, 5, 6]])
```

```python
df2 = pd.concat([df, df])
```

```python
df2.reset_index()
#Output
#index  0  1  2  3
#0      0  1  2  3  4
#1      1  2  3  4  5
#2      2  3  4  5  6
#3      0  1  2  3  4
#4      1  2  3  4  5
#5      2  3  4  5  6
#```
#How can I reset_index without adding a new column index?
```

Response: You can use the drop=True option in reset_index(). See here.


Question:
Set value on an entire column of a pandas dataframe I'm trying to set the entire column of a dataframe to a specific value.
```
In  [1]: df
Out [1]:
     issueid   industry
0        001        xxx
1        002        xxx
2        003        xxx
3        004        xxx
4        005        xxx
```
From what I've seen, loc is the best practice when replacing values in a dataframe (or isn't it?):
```
In  [2]: df.loc[:,'industry'] = 'yyy'
```
However, I still received this much talked-about warning message:
```
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
```
If I do
```
In  [3]: df['industry'] = 'yyy'
```
I got the same warning message. Any ideas? Working with Python 3.5.2 and pandas 0.18.1. EDIT Jan 2023: Given the volume of visits on this question, it's worth stating that my original question was really more about dataframe copy-versus-slice than "setting value to an entire column". On copy-versus-slice: My current understanding is that, in general, if you want to modify a subset of a dataframe after slicing, you should create the subset by .copy(). If you only want a view of the slice, no copy() needed. On setting value to an entire column: simply do df[col_name] = col_value

Response:
You can use the assign function:
```
df = df.assign(industry='yyy')
```


Question:
Check if a value exists in pandas dataframe index I am sure there is an obvious way to do this but cant think of anything slick right now. Basically instead of raising exception I would like to get True or False to see if a value exists in pandas df index.
```
import pandas as pd
df = pd.DataFrame({'test':[1,2,3,4]}, index=['a','b','c','d'])
df.loc['g']  # (should give False)
```
What I have working now is the following
```
sum(df.index == 'g')
```

Response:
This should do the trick
```
'g' in df.index
```


Question: How can I check if a Pandas dataframe's index is sorted I have a vanilla pandas dataframe with an index. I need to check if the index is sorted. Preferably without sorting it again. e.g. I can test an index to see if it is unique by index.is_unique() is there a similar way for testing sorted?

Response: How about: df.index.is_monotonic


Question:
First non-null value per row from a list of Pandas columns If I've got a DataFrame in pandas which looks something like:
```
A   B   C
0   1 NaN   2
1 NaN   3 NaN
2 NaN   4   5
3 NaN NaN NaN
```
How can I get the first non-null value from each row? E.g. for the above, I'd like to get: [1, 3, 4, None] (or equivalent Series).

Response:
Fill the nans from the left with fillna, then get the leftmost column:
```
df.fillna(method='bfill', axis=1).iloc[:, 0]
```


Question:
Pandas Split Dataframe into two Dataframes at a specific column I have pandas DataFrame which I have composed from concat. One row consists of 96 values, I would like to split the DataFrame from the value 72. So that the first 72 values of a row are stored in Dataframe1, and the next 24 values of a row in Dataframe2. I create my DF as follows:
```
temps = DataFrame(myData)
datasX = concat(
[temps.shift(72), temps.shift(71), temps.shift(70), temps.shift(69), temps.shift(68), temps.shift(67),
 temps.shift(66), temps.shift(65), temps.shift(64), temps.shift(63), temps.shift(62), temps.shift(61),
 temps.shift(60), temps.shift(59), temps.shift(58), temps.shift(57), temps.shift(56), temps.shift(55),
 temps.shift(54), temps.shift(53), temps.shift(52), temps.shift(51), temps.shift(50), temps.shift(49),
 temps.shift(48), temps.shift(47), temps.shift(46), temps.shift(45), temps.shift(44), temps.shift(43),
 temps.shift(42), temps.shift(41), temps.shift(40), temps.shift(39), temps.shift(38), temps.shift(37),
 temps.shift(36), temps.shift(35), temps.shift(34), temps.shift(33), temps.shift(32), temps.shift(31),
 temps.shift(30), temps.shift(29), temps.shift(28), temps.shift(27), temps.shift(26), temps.shift(25),
 temps.shift(24), temps.shift(23), temps.shift(22), temps.shift(21), temps.shift(20), temps.shift(19),
 temps.shift(18), temps.shift(17), temps.shift(16), temps.shift(15), temps.shift(14), temps.shift(13),
 temps.shift(12), temps.shift(11), temps.shift(10), temps.shift(9), temps.shift(8), temps.shift(7),
 temps.shift(6), temps.shift(5), temps.shift(4), temps.shift(3), temps.shift(2), temps.shift(1), temps,
 temps.shift(-1), temps.shift(-2), temps.shift(-3), temps.shift(-4), temps.shift(-5), temps.shift(-6),
 temps.shift(-7), temps.shift(-8), temps.shift(-9), temps.shift(-10), temps.shift(-11), temps.shift(-12),
 temps.shift(-13), temps.shift(-14), temps.shift(-15), temps.shift(-16), temps.shift(-17), temps.shift(-18),
 temps.shift(-19), temps.shift(-20), temps.shift(-21), temps.shift(-22), temps.shift(-23)], axis=1)
```
Question is: How can split them? :)

Response:
iloc
```
df1 = datasX.iloc[:, :72]
df2 = datasX.iloc[:, 72:]
```
(iloc docs)


Question:
How to get/set a pandas index column title or name? How do I get the index column name in Python's pandas? Here's an example dataframe:
```none
Column 1
Index Title
Apples              1
Oranges             2
Puppies             3
Ducks               4
```
What I'm trying to do is get/set the dataframe's index title. Here is what I tried:
```py
import pandas as pd

data = {'Column 1'   : [1., 2., 3., 4.],
        'Index Title': ["Apples", "Oranges", "Puppies", "Ducks"]}
df = pd.DataFrame(data)
df.index = df["Index Title"]
del df["Index Title"]
```
Anyone know how to do this?

Response:
You can just get/set the index via its name property
```

```python
df.index.name
#Output
#'Index Title'
```

```python
df.index.name = 'foo'
```

```python
df.index.name
#Output
#'foo'
```

```python
df
#Output
#Column 1
#foo
#Apples          1
#Oranges         2
#Puppies         3
#Ducks           4
#```
```


Question:
Filter dataframe rows if value in column is in a set list of values [duplicate] This question already has answers here: How to filter Pandas dataframe using 'in' and 'not in' like in SQL (12 answers) Use a list of values to select rows from a Pandas dataframe (9 answers) Closed 6 years ago. I have a Python pandas DataFrame rpt:
```
rpt
<class 'pandas.core.frame.DataFrame'>
MultiIndex: 47518 entries, ('000002', '20120331') to ('603366', '20091231')
Data columns:
STK_ID                    47518  non-null values
STK_Name                  47518  non-null values
RPT_Date                  47518  non-null values
sales                     47518  non-null values
```
I can filter the rows whose stock id is '600809' like this: rpt[rpt['STK_ID'] == '600809']
```
<class 'pandas.core.frame.DataFrame'>
MultiIndex: 25 entries, ('600809', '20120331') to ('600809', '20060331')
Data columns:
STK_ID                    25  non-null values
STK_Name                  25  non-null values
RPT_Date                  25  non-null values
sales                     25  non-null values
```
and I want to get all the rows of some stocks together, such as ['600809','600141','600329']. That means I want a syntax like this:
```
stk_list = ['600809','600141','600329']

rst = rpt[rpt['STK_ID'] in stk_list] # this does not works in pandas
```
Since pandas not accept above command, how to achieve the target?

Response: Use the isin method: rpt[rpt['STK_ID'].isin(stk_list)]


Question:
Pandas sum across columns and divide each cell from that value I have read a csv file and pivoted it to get to following structure:
```
pivoted = df.pivot('user_id', 'group', 'value')
lookup = df.drop_duplicates('user_id')[['user_id', 'group']]
lookup.set_index(['user_id'], inplace=True)
result = pivoted.join(lookup)
result = result.fillna(0)
```
Section of the result:
```
0     1     2    3     4    5   6  7    8   9  10  11  12  13  group
user_id
2        33653  2325   916  720   867  187  31  0    6   3  42  56  92  15    l-1
4        18895   414  1116  570  1190   55  92  0  122  23  78   6   4   2    l-2
16        1383    70    27   17    17    1   0  0    0   0   1   0   0   0    l-2
50         396    72    34    5    18    0   0  0    0   0   0   0   0   0    l-3
51        3915  1170   402  832  2791  316  12  5  118  51  32   9  62  27    l-4
```
I want to sum across column 0 to column 13 by each row and divide each cell by the sum of that row. I am still getting used to pandas; if I understand correctly, we should try to avoid for loops when doing things like this? In other words, how can I do this in a 'pandas' way?

Response:
More simply:
```
result.div(result.sum(axis=1), axis=0)
```


Question:
Lambda including if...elif...else I want to apply a lambda function to a DataFrame column using if...elif...else within the lambda function. The df and the code are something like:
```
df=pd.DataFrame({"one":[1,2,3,4,5],"two":[6,7,8,9,10]})

df["one"].apply(lambda x: x*10 if x<2 elif x<4 x**2 else x+10)
```
Obviously, this doesn't work. Is there a way to apply if....elif....else to a lambda? How can I get the same result with List Comprehension?

Response:
Nest if .. elses:
```
lambda x: x*10 if x<2 else (x**2 if x<4 else x+10)
```


Question:
Normalize DataFrame by group Let's say that I have some data generated as follows:
```
N = 20
m = 3
data = np.random.normal(size=(N,m)) + np.random.normal(size=(N,m))**3
```
and then I create some categorization variable:
```
indx = np.random.randint(0,3,size=N).astype(np.int32)
```
and generate a DataFrame:
```
import pandas as pd
df = pd.DataFrame(np.hstack((data, indx[:,None])),
             columns=['a%s' % k for k in range(m)] + [ 'indx'])
```
I can get the mean value, per group as:
```
df.groubpy('indx').mean()
```
What I'm unsure of how to do is to then subtract the mean off of each group, per-column in the original data, so that the data in each column is normalized by the mean within group. Any suggestions would be appreciated.

Response:
```

```python
df.groupby('indx').transform(lambda x: (x - x.mean()) / x.std())
```
should do it.
```


Question: Python - Dimension of Data Frame New to Python. In R, you can get the dimension of a matrix using dim(...). What is the corresponding function in Python Pandas for their data frame?

Response: df.shape, where df is your DataFrame.


Question:
Convert list of dictionaries to a pandas DataFrame How can I convert a list of dictionaries into a DataFrame? I want to turn
```py
[{'points': 50, 'time': '5:00', 'year': 2010},
 {'points': 25, 'time': '6:00', 'month': "february"},
 {'points':90, 'time': '9:00', 'month': 'january'},
 {'points_h1':20, 'month': 'june'}]
```
into
```none
month  points  points_h1  time  year
0       NaN      50        NaN  5:00  2010
1  february      25        NaN  6:00   NaN
2   january      90        NaN  9:00   NaN
3      june     NaN         20   NaN   NaN
```

Response:
If ds is a list of dicts:
```
df = pd.DataFrame(ds)
```
Note: this does not work with nested data.


Question:
Why do we need to call zero_grad() in PyTorch? Why does zero_grad() need to be called during training?
```
|  zero_grad(self)
|      Sets gradients of all model parameters to zero.
```

Response:
In PyTorch, for every mini-batch during the training phase, we typically want to explicitly set the gradients to zero before starting to do backpropagation (i.e., updating the Weights and biases) because PyTorch accumulates the gradients on subsequent backward passes. This accumulating behavior is convenient while training RNNs or when we want to compute the gradient of the loss summed over multiple mini-batches. So, the default action has been set to accumulate (i.e. sum) the gradients on every loss.backward() call. Because of this, when you start your training loop, ideally you should zero out the gradients so that you do the parameter update correctly. Otherwise, the gradient would be a combination of the old gradient, which you have already used to update your model parameters and the newly-computed gradient. It would therefore point in some other direction than the intended direction towards the minimum (or maximum, in case of maximization objectives). Here is a simple example:
```
import torch
from torch.autograd import Variable
import torch.optim as optim

def linear_model(x, W, b):
    return torch.matmul(x, W) + b

data, targets = ...

W = Variable(torch.randn(4, 3), requires_grad=True)
b = Variable(torch.randn(3), requires_grad=True)

optimizer = optim.Adam([W, b])

for sample, target in zip(data, targets):
    # clear out the gradients of all Variables
    # in this optimizer (i.e. W, b)
    optimizer.zero_grad()
    output = linear_model(sample, W, b)
    loss = (output - target) ** 2
    loss.backward()
    optimizer.step()
```
Alternatively, if you're doing a vanilla gradient descent, then:
```
W = Variable(torch.randn(4, 3), requires_grad=True)
b = Variable(torch.randn(3), requires_grad=True)

for sample, target in zip(data, targets):
    # clear out the gradients of Variables
    # (i.e. W, b)
    W.grad.data.zero_()
    b.grad.data.zero_()

    output = linear_model(sample, W, b)
    loss = (output - target) ** 2
    loss.backward()

    W -= learning_rate * W.grad.data
    b -= learning_rate * b.grad.data
```
Note: The accumulation (i.e., sum) of gradients happens when .backward() is called on the loss tensor. As of v1.7.0, Pytorch offers the option to reset the gradients to None optimizer.zero_grad(set_to_none=True) instead of filling them with a tensor of zeroes. The docs claim that this setting reduces memory requirements and slightly improves performance, but might be error-prone if not handled carefully.


Question:
How do I print the model summary in PyTorch? How do I print the summary of a model in PyTorch like what model.summary() does in Keras:
```
Model Summary:
____________________________________________________________________________________________________
Layer (type)                     Output Shape          Param #     Connected to
====================================================================================================
input_1 (InputLayer)             (None, 1, 15, 27)     0
____________________________________________________________________________________________________
convolution2d_1 (Convolution2D)  (None, 8, 15, 27)     872         input_1[0][0]
____________________________________________________________________________________________________
maxpooling2d_1 (MaxPooling2D)    (None, 8, 7, 27)      0           convolution2d_1[0][0]
____________________________________________________________________________________________________
flatten_1 (Flatten)              (None, 1512)          0           maxpooling2d_1[0][0]
____________________________________________________________________________________________________
dense_1 (Dense)                  (None, 1)             1513        flatten_1[0][0]
====================================================================================================
Total params: 2,385
Trainable params: 2,385
Non-trainable params: 0
```

Response:
Yes, you can get exact Keras representation, using the pytorch-summary package. Example for VGG16:
```
from torchvision import models
from torchsummary import summary

vgg = models.vgg16()
summary(vgg, (3, 224, 224))

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1         [-1, 64, 224, 224]           1,792
              ReLU-2         [-1, 64, 224, 224]               0
            Conv2d-3         [-1, 64, 224, 224]          36,928
              ReLU-4         [-1, 64, 224, 224]               0
         MaxPool2d-5         [-1, 64, 112, 112]               0
            Conv2d-6        [-1, 128, 112, 112]          73,856
              ReLU-7        [-1, 128, 112, 112]               0
            Conv2d-8        [-1, 128, 112, 112]         147,584
              ReLU-9        [-1, 128, 112, 112]               0
        MaxPool2d-10          [-1, 128, 56, 56]               0
           Conv2d-11          [-1, 256, 56, 56]         295,168
             ReLU-12          [-1, 256, 56, 56]               0
           Conv2d-13          [-1, 256, 56, 56]         590,080
             ReLU-14          [-1, 256, 56, 56]               0
           Conv2d-15          [-1, 256, 56, 56]         590,080
             ReLU-16          [-1, 256, 56, 56]               0
        MaxPool2d-17          [-1, 256, 28, 28]               0
           Conv2d-18          [-1, 512, 28, 28]       1,180,160
             ReLU-19          [-1, 512, 28, 28]               0
           Conv2d-20          [-1, 512, 28, 28]       2,359,808
             ReLU-21          [-1, 512, 28, 28]               0
           Conv2d-22          [-1, 512, 28, 28]       2,359,808
             ReLU-23          [-1, 512, 28, 28]               0
        MaxPool2d-24          [-1, 512, 14, 14]               0
           Conv2d-25          [-1, 512, 14, 14]       2,359,808
             ReLU-26          [-1, 512, 14, 14]               0
           Conv2d-27          [-1, 512, 14, 14]       2,359,808
             ReLU-28          [-1, 512, 14, 14]               0
           Conv2d-29          [-1, 512, 14, 14]       2,359,808
             ReLU-30          [-1, 512, 14, 14]               0
        MaxPool2d-31            [-1, 512, 7, 7]               0
           Linear-32                 [-1, 4096]     102,764,544
             ReLU-33                 [-1, 4096]               0
          Dropout-34                 [-1, 4096]               0
           Linear-35                 [-1, 4096]      16,781,312
             ReLU-36                 [-1, 4096]               0
          Dropout-37                 [-1, 4096]               0
           Linear-38                 [-1, 1000]       4,097,000
================================================================
Total params: 138,357,544
Trainable params: 138,357,544
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.57
Forward/backward pass size (MB): 218.59
Params size (MB): 527.79
Estimated Total Size (MB): 746.96
----------------------------------------------------------------
```


Question:
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same This:
```
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

for data in dataloader:
    inputs, labels = data
    outputs = model(inputs)
```
Gives the error: RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

Response:
You get this error because your model is on the GPU, but your data is on the CPU. So, you need to send your input tensors to the GPU.
```py
inputs, labels = data                         # this is what you had
inputs, labels = inputs.cuda(), labels.cuda() # add this line
```
Or like this, to stay consistent with the rest of your code:
```py
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

inputs, labels = inputs.to(device), labels.to(device)
```
The same error will be raised if your input tensors are on the GPU but your model weights aren't. In this case, you need to send your model weights to the GPU.
```
model = MyModel()

if torch.cuda.is_available():
    model.cuda()
```
See the documentation for cuda(), and its opposite, cpu().


Question:
How to avoid "CUDA out of memory" in PyTorch I think it's a pretty common message for PyTorch users with low GPU memory:
```
RuntimeError: CUDA out of memory. Tried to allocate X MiB (GPU X; X GiB total capacity; X GiB already allocated; X MiB free; X cached)
```
I tried to process an image by loading each layer to GPU and then loading it back:
```py
for m in self.children():
    m.cuda()
    x = m(x)
    m.cpu()
    torch.cuda.empty_cache()
```
But it doesn't seem to be very effective. I'm wondering is there any tips and tricks to train large deep learning models while using little GPU memory.

Response:
Although
```
import torch
torch.cuda.empty_cache()
```
provides a good alternative for clearing the occupied cuda memory and we can also manually clear the not in use variables by using,
```
import gc
del variables
gc.collect()
```
But still after using these commands, the error might appear again because pytorch doesn't actually clears the memory instead clears the reference to the memory occupied by the variables. So reducing the batch_size after restarting the kernel and finding the optimum batch_size is the best possible option (but sometimes not a very feasible one). Another way to get a deeper insight into the alloaction of memory in gpu is to use:
```
torch.cuda.memory_summary(device=None, abbreviated=False)
```
wherein, both the arguments are optional. This gives a readable summary of memory allocation and allows you to figure the reason of CUDA running out of memory and restart the kernel to avoid the error from happening again (Just like I did in my case). Passing the data iteratively might help but changing the size of layers of your network or breaking them down would also prove effective (as sometimes the model also occupies a significant memory for example, while doing transfer learning).


Question:
Pytorch tensor to numpy array I have a pytorch Tensor of shape [4, 3, 966, 1296]. I want to convert it to numpy array using the following code:
```
imgs = imgs.numpy()[:, ::-1, :, :]
```
How does that code work?

Response:
I believe you also have to use .detach(). I had to convert my Tensor to a numpy array on Colab which uses CUDA and GPU. I did it like the following:
```
# this is just my embedding matrix which is a Torch tensor object
embedding = learn.model.u_weight

embedding_list = list(range(0, 64382))

input = torch.cuda.LongTensor(embedding_list)
tensor_array = embedding(input)
# the output of the line below is a numpy array
tensor_array.cpu().detach().numpy()
```


Question:
Pytorch, what are the gradient arguments I am reading through the documentation of PyTorch and found an example where they write
```
gradients = torch.FloatTensor([0.1, 1.0, 0.0001])
y.backward(gradients)
print(x.grad)
```
where x was an initial variable, from which y was constructed (a 3-vector). The question is, what are the 0.1, 1.0 and 0.0001 arguments of the gradients tensor ? The documentation is not very clear on that.

Response:
Explanation For neural networks, we usually use loss to assess how well the network has learned to classify the input image (or other tasks). The loss term is usually a scalar value. In order to update the parameters of the network, we need to calculate the gradient of loss w.r.t to the parameters, which is actually leaf node in the computation graph (by the way, these parameters are mostly the weight and bias of various layers such Convolution, Linear and so on). According to chain rule, in order to calculate gradient of loss w.r.t to a leaf node, we can compute derivative of loss w.r.t some intermediate variable, and gradient of intermediate variable w.r.t to the leaf variable, do a dot product and sum all these up. The gradient arguments of a Variable's backward() method is used to calculate a weighted sum of each element of a Variable w.r.t the leaf Variable. These weight is just the derivate of final loss w.r.t each element of the intermediate variable. A concrete example Let's take a concrete and simple example to understand this.
```py
from torch.autograd import Variable
import torch
x = Variable(torch.FloatTensor([[1, 2, 3, 4]]), requires_grad=True)
z = 2*x
loss = z.sum(dim=1)

# do backward for first element of z
z.backward(torch.FloatTensor([[1, 0, 0, 0]]), retain_graph=True)
print(x.grad.data)
x.grad.data.zero_() #remove gradient in x.grad, or it will be accumulated

# do backward for second element of z
z.backward(torch.FloatTensor([[0, 1, 0, 0]]), retain_graph=True)
print(x.grad.data)
x.grad.data.zero_()

# do backward for all elements of z, with weight equal to the derivative of
# loss w.r.t z_1, z_2, z_3 and z_4
z.backward(torch.FloatTensor([[1, 1, 1, 1]]), retain_graph=True)
print(x.grad.data)
x.grad.data.zero_()

# or we can directly backprop using loss
loss.backward() # equivalent to loss.backward(torch.FloatTensor([1.0]))
print(x.grad.data)
```
In the above example, the outcome of first print is 2 0 0 0 [torch.FloatTensor of size 1x4] which is exactly the derivative of z_1 w.r.t to x. The outcome of second print is : 0 2 0 0 [torch.FloatTensor of size 1x4] which is the derivative of z_2 w.r.t to x. Now if use a weight of [1, 1, 1, 1] to calculate the derivative of z w.r.t to x, the outcome is 1*dz_1/dx + 1*dz_2/dx + 1*dz_3/dx + 1*dz_4/dx. So no surprisingly, the output of 3rd print is: 2 2 2 2 [torch.FloatTensor of size 1x4] It should be noted that weight vector [1, 1, 1, 1] is exactly derivative of loss w.r.t to z_1, z_2, z_3 and z_4. The derivative of loss w.r.t to x is calculated as:
```
d(loss)/dx = d(loss)/dz_1 * dz_1/dx + d(loss)/dz_2 * dz_2/dx + d(loss)/dz_3 * dz_3/dx + d(loss)/dz_4 * dz_4/dx
```
So the output of 4th print is the same as the 3rd print: 2 2 2 2 [torch.FloatTensor of size 1x4]


Question:
What does "unsqueeze" do in Pytorch? The PyTorch documentation says: Returns a new tensor with a dimension of size one inserted at the specified position. [...]
```

```python
x = torch.tensor([1, 2, 3, 4])
```

```python
torch.unsqueeze(x, 0)
#Output
#tensor([[ 1,  2,  3,  4]])
```

```python
torch.unsqueeze(x, 1)
#Output
#tensor([[ 1],
#        [ 2],
#        [ 3],
#        [ 4]])
#```
```

Response: unsqueeze turns an n.d. tensor into an (n+1).d. one by adding an extra dimension of depth 1. However, since it is ambiguous which axis the new dimension should lie across (i.e. in which direction it should be "unsqueezed"), this needs to be specified by the dim argument. e.g. unsqueeze can be applied to a 2d tensor three different ways: The resulting unsqueezed tensors have the same information, but the indices used to access them are different.


Question: Understanding `torch.nn.Parameter()` How does torch.nn.Parameter() work?

Response:
I will break it down for you. Tensors, as you might know, are multi dimensional matrices. Parameter, in its raw form, is a tensor i.e. a multi dimensional matrix. It sub-classes the Variable class. The difference between a Variable and a Parameter comes in when associated with a module. When a Parameter is associated with a module as a model attribute, it gets added to the parameter list automatically and can be accessed using the 'parameters' iterator. Initially in Torch, a Variable (which could for example be an intermediate state) would also get added as a parameter of the model upon assignment. Later on there were use cases identified where a need to cache the variables instead of having them added to the parameter list was identified. One such case, as mentioned in the documentation is that of RNN, where in you need to save the last hidden state so you don't have to pass it again and again. The need to cache a Variable instead of having it automatically register as a parameter to the model is why we have an explicit way of registering parameters to our model i.e. nn.Parameter class. For instance, run the following code -
```
import torch
import torch.nn as nn
from torch.optim import Adam

class NN_Network(nn.Module):
    def __init__(self,in_dim,hid,out_dim):
        super(NN_Network, self).__init__()
        self.linear1 = nn.Linear(in_dim,hid)
        self.linear2 = nn.Linear(hid,out_dim)
        self.linear1.weight = torch.nn.Parameter(torch.zeros(in_dim,hid))
        self.linear1.bias = torch.nn.Parameter(torch.ones(hid))
        self.linear2.weight = torch.nn.Parameter(torch.zeros(in_dim,hid))
        self.linear2.bias = torch.nn.Parameter(torch.ones(hid))

    def forward(self, input_array):
        h = self.linear1(input_array)
        y_pred = self.linear2(h)
        return y_pred

in_d = 5
hidn = 2
out_d = 3
net = NN_Network(in_d, hidn, out_d)
```
Now, check the parameter list associated with this model -
```
for param in net.parameters():
    print(type(param.data), param.size())

""" Output
<class 'torch.FloatTensor'> torch.Size([5, 2])
<class 'torch.FloatTensor'> torch.Size([2])
<class 'torch.FloatTensor'> torch.Size([5, 2])
<class 'torch.FloatTensor'> torch.Size([2])
"""
```
Or try,
```
list(net.parameters())
```
This can easily be fed to your optimizer -
```
opt = Adam(net.parameters(), learning_rate=0.001)
```
Also, note that Parameters have require_grad set by default.


Question: L1/L2 regularization in PyTorch How do I add L1/L2 regularization in PyTorch without manually computing it?

Response:
Use weight_decay > 0 for L2 regularization:
```
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4, weight_decay=1e-5)
```


Question:
How to fix RuntimeError "Expected object of scalar type Float but got scalar type Double for argument"? I'm trying to train a classifier via PyTorch. However, I am experiencing problems with training when I feed the model with training data. I get this error on y_pred = model(X_trainTensor): RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 'mat1' Here are key parts of my code:
```py
# Hyper-parameters
D_in = 47  # there are 47 parameters I investigate
H = 33
D_out = 2  # output should be either 1 or 0
```
```py
# Format and load the data
y = np.array( df['target'] )
X = np.array( df.drop(columns = ['target'], axis = 1) )
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size = 0.8)  # split training/test data

X_trainTensor = torch.from_numpy(X_train) # convert to tensors
y_trainTensor = torch.from_numpy(y_train)
X_testTensor = torch.from_numpy(X_test)
y_testTensor = torch.from_numpy(y_test)
```
```py
# Define the model
model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
    nn.LogSoftmax(dim = 1)
)
```
```py
# Define the loss function
loss_fn = torch.nn.NLLLoss()
```
```py
for i in range(50):
    y_pred = model(X_trainTensor)
    loss = loss_fn(y_pred, y_trainTensor)
    model.zero_grad()
    loss.backward()
    with torch.no_grad():
        for param in model.parameters():
            param -= learning_rate * param.grad
```

Response: Reference is from this github issue. When the error is RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 'mat1', you would need to use the .float() function since it says Expected object of scalar type Float. Therefore, the solution is changing y_pred = model(X_trainTensor) to y_pred = model(X_trainTensor.float()). Likewise, when you get another error for loss = loss_fn(y_pred, y_trainTensor), you need y_trainTensor.long() since the error message says Expected object of scalar type Long. You could also do model.double(), as suggested by @Paddy .


Question: How does the "number of workers" parameter in PyTorch dataloader actually work? If num_workers is 2, Does that mean that it will put 2 batches in the RAM and send 1 of them to the GPU or Does it put 3 batches in the RAM then sends 1 of them to the GPU? What does actually happen when the number of workers is higher than the number of CPU cores? I tried it and it worked fine but How does it work? (I thought that the maximum number of workers I can choose is the number of cores). If I set num_workers to 3 and during the training there were no batches in the memory for the GPU, Does the main process waits for its workers to read the batches or Does it read a single batch (without waiting for the workers)?

Response: When num_workers>0, only these workers will retrieve data, main process won't. So when num_workers=2 you have at most 2 workers simultaneously putting data into RAM, not 3. Well our CPU can usually run like 100 processes without trouble and these worker processes aren't special in anyway, so having more workers than cpu cores is ok. But is it efficient? it depends on how busy your cpu cores are for other tasks, speed of cpu, speed of your hard disk etc. In short, its complicated, so setting workers to number of cores is a good rule of thumb, nothing more. Nope. Remember DataLoader doesn't just randomly return from what's available in RAM right now, it uses batch_sampler to decide which batch to return next. Each batch is assigned to a worker, and main process will wait until the desired batch is retrieved by assigned worker. Lastly to clarify, it isn't DataLoader's job to send anything directly to GPU, you explicitly call cuda() for that. EDIT: Don't call cuda() inside Dataset's __getitem__() method, please look at @psarka's comment for the reasoning


Question: Convert PyTorch tensor to python list How do I convert a PyTorch Tensor into a python list? I want to convert a tensor of size [1, 2048, 1, 1] into a list of 2048 elements. My tensor has floating point values. Is there a solution which also works with other data types such as int?

Response:
Use Tensor.tolist() e.g:
```

```python
import torch
```

```python
a = torch.randn(2, 2)
```

```python
a.tolist()
#Output
#[[0.012766935862600803, 0.5415473580360413],
# [-0.08909505605697632, 0.7729271650314331]]
```

```python
a[0,0].tolist()
#Output
#0.012766935862600803
#```
#To remove all dimensions of size 1, use a.squeeze().tolist(). Alternatively, if all but one dimension are of size 1 (or you wish to get a list of every element of the tensor) you may use a.flatten().tolist().
```


Question:
How do I convert a Pandas dataframe to a PyTorch tensor? How do I train a simple neural network with PyTorch on a pandas dataframe df? The column df["Target"] is the target (e.g. labels) of the network. This doesn't work:
```
import pandas as pd
import torch.utils.data as data_utils

target = pd.DataFrame(df['Target'])
train = data_utils.TensorDataset(df, target)
train_loader = data_utils.DataLoader(train, batch_size=10, shuffle=True)
```

Response:
I'm referring to the question in the title as you haven't really specified anything else in the text, so just converting the DataFrame into a PyTorch tensor. Without information about your data, I'm just taking float values as example targets here. Convert Pandas dataframe to PyTorch tensor?
```py
import pandas as pd
import torch
import random

# creating dummy targets (float values)
targets_data = [random.random() for i in range(10)]

# creating DataFrame from targets_data
targets_df = pd.DataFrame(data=targets_data)
targets_df.columns = ['targets']

# creating tensor from targets_df
torch_tensor = torch.tensor(targets_df['targets'].values)

# printing out result
print(torch_tensor)
```
Output:
```
tensor([ 0.5827,  0.5881,  0.1543,  0.6815,  0.9400,  0.8683,  0.4289,
         0.5940,  0.6438,  0.7514], dtype=torch.float64)
```
Tested with Pytorch 0.4.0. I hope this helps, if you have any further questions - just ask. :)


Question:
Data Augmentation in PyTorch I am a little bit confused about the data augmentation performed in PyTorch. Now, as far as I know, when we are performing data augmentation, we are KEEPING our original dataset, and then adding other versions of it (Flipping, Cropping...etc). But that doesn't seem like happening in PyTorch. As far as I understood from the references, when we use data.transforms in PyTorch, then it applies them one by one. So for example:
```python
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}
```
Here , for the training, we are first randomly cropping the image and resizing it to shape (224,224). Then we are taking these (224,224) images and horizontally flipping them. Therefore, our dataset is now containing ONLY the horizontally flipped images, so our original images are lost in this case. Am I right? Is this understanding correct? If not, then where do we tell PyTorch in this code above (taken from Official Documentation) to keep the original images and resize them to the expected shape (224,224)?

Response:
I assume you are asking whether these data augmentation transforms (e.g. RandomHorizontalFlip) actually increase the size of the dataset as well, or are they applied on each item in the dataset one by one and not adding to the size of the dataset. Running the following simple code snippet we could observe that the latter is true, i.e. if you have a dataset of 8 images, and create a PyTorch dataset object for this dataset when you iterate through the dataset, the transformations are called on each data point, and the transformed data point is returned. So for example if you have random flipping, some of the data points are returned as original, some are returned as flipped (e.g. 4 flipped and 4 original). In other words, by one iteration through the dataset items, you get 8 data points(some flipped and some not). [Which is at odds with the conventional understanding of augmenting the dataset(e.g. in this case having 16 data points in the augmented dataset)]
```
from torch.utils.data import Dataset
from torchvision import transforms

class experimental_dataset(Dataset):

    def __init__(self, data, transform):
        self.data = data
        self.transform = transform

    def __len__(self):
        return len(self.data.shape[0])

    def __getitem__(self, idx):
        item = self.data[idx]
        item = self.transform(item)
        return item

transform = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor()
])

x = torch.rand(8, 1, 2, 2)
print(x)

dataset = experimental_dataset(x,transform)

for item in dataset:
    print(item)
```
Results: (The little differences in floating points are caused by transforming to pil image and back) Original dummy dataset:
```
tensor([[[[0.1872, 0.5518],
          [0.5733, 0.6593]]],

    [[[0.6570, 0.6487],
      [0.4415, 0.5883]]],

    [[[0.5682, 0.3294],
      [0.9346, 0.1243]]],

    [[[0.1829, 0.5607],
      [0.3661, 0.6277]]],

    [[[0.1201, 0.1574],
      [0.4224, 0.6146]]],

    [[[0.9301, 0.3369],
      [0.9210, 0.9616]]],

    [[[0.8567, 0.2297],
      [0.1789, 0.8954]]],

    [[[0.0068, 0.8932],
      [0.9971, 0.3548]]]])
```
transformed dataset:
```
tensor([[[0.1843, 0.5490],
     [0.5725, 0.6588]]])
tensor([[[0.6549, 0.6471],
     [0.4392, 0.5882]]])
tensor([[[0.5647, 0.3255],
         [0.9333, 0.1216]]])
tensor([[[0.5569, 0.1804],
         [0.6275, 0.3647]]])
tensor([[[0.1569, 0.1176],
         [0.6118, 0.4196]]])
tensor([[[0.9294, 0.3333],
         [0.9176, 0.9608]]])
tensor([[[0.8549, 0.2275],
         [0.1765, 0.8941]]])
tensor([[[0.8902, 0.0039],
         [0.3529, 0.9961]]])
```


Question:
How to change the learning rate of an optimizer at any given moment (no LR schedule)? Is it possible in PyTorch to change the learning rate of the optimizer in the middle of training dynamically (I don't want to define a learning rate schedule beforehand)? So let's say I have an optimizer:
```
optim = torch.optim.SGD(model.parameters(), lr=0.01)
```
Now due to some tests which I perform during training, I realize my learning rate is too high so I want to change it to say 0.001. There doesn't seem to be a method optim.set_lr(0.001) but is there some way to do this?

Response:
So the learning rate is stored in optim.param_groups[i]['lr']. optim.param_groups is a list of the different weight groups which can have different learning rates. Thus, simply doing:
```
for g in optim.param_groups:
    g['lr'] = 0.001
```
will do the trick. **Alternatively**, as mentionned in the comments, if your learning rate only depends on the epoch number, you can use a learning rate scheduler. For example (modified example from the doc):
```
from torch.optim.lr_scheduler import LambdaLR
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
# Assuming the optimizer has two groups.
lambda_group1 = lambda epoch: epoch // 30
lambda_group2 = lambda epoch: 0.95 ** epoch
scheduler = LambdaLR(optimizer, lr_lambda=[lambda1, lambda2])
for epoch in range(100):
    train(...)
    validate(...)
    scheduler.step()
```
Also, there is a prebuilt learning rate scheduler to reduce on plateaus.


Question: PyTorch: How to get the shape of a Tensor as a list of int In numpy, V.shape gives a tuple of ints of dimensions of V. In tensorflow V.get_shape().as_list() gives a list of integers of the dimensions of V. In pytorch, V.size() gives a size object, but how do I convert it to ints?

Response:
For PyTorch v1.0 and possibly above:
```

```python
import torch
```

```python
var = torch.tensor([[1,0], [0,1]])
```

# Using .size function, returns a torch.Size object.

```python
var.size()
#Output
#torch.Size([2, 2])
```

```python
type(var.size())
#Output
#<class 'torch.Size'>
```

# Similarly, using .shape

```python
var.shape
#Output
#torch.Size([2, 2])
```

```python
type(var.shape)
#Output
#<class 'torch.Size'>
#```
#You can cast any torch.Size object to a native Python list:
#```
```

```python
list(var.size())
#Output
#[2, 2]
```

```python
type(list(var.size()))
#Output
#<class 'list'>
#```
#In PyTorch v0.3 and 0.4: Simply list(var.size()), e.g.:
#```
```

```python
import torch
```

```python
from torch.autograd import Variable
```

```python
from torch import IntTensor
```

```python
var = Variable(IntTensor([[1,0],[0,1]]))
```

```python
var
#Output
#Variable containing:
# 1  0
# 0  1
#[torch.IntTensor of size 2x2]
```

```python
var.size()
#Output
#torch.Size([2, 2])
```

```python
list(var.size())
#Output
#[2, 2]
#```
```


Question:
How to clear CUDA memory in PyTorch I am trying to get the output of a neural network which I have already trained. The input is an image of the size 300x300. I am using a batch size of 1, but I still get a CUDA error: out of memory error after I have successfully got the output for 25 images. I tried torch.cuda.empty_cache(), but this still doesn't seem to solve the problem. Code:
```
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

train_x = torch.tensor(train_x, dtype=torch.float32).view(-1, 1, 300, 300)
train_x = train_x.to(device)
dataloader = torch.utils.data.DataLoader(train_x, batch_size=1, shuffle=False)

right = []
for i, left in enumerate(dataloader):
    print(i)
    temp = model(left).view(-1, 1, 300, 300)
    right.append(temp.to('cpu'))
    del temp
    torch.cuda.empty_cache()
```
This for loop runs for 25 times every time before giving the memory error. Every time, I am sending a new image in the network for computation. So, I don't really need to store the previous computation results in the GPU after every iteration in the loop. Is there any way to achieve this?

Response:
I figured out where I was going wrong. I am posting the solution as an answer for others who might be struggling with the same problem. Basically, what PyTorch does is that it creates a computational graph whenever I pass the data through my network and stores the computations on the GPU memory, in case I want to calculate the gradient during backpropagation. But since I only wanted to perform a forward propagation, I simply needed to specify torch.no_grad() for my model. Thus, the for loop in my code could be rewritten as:
```
for i, left in enumerate(dataloader):
    print(i)
    with torch.no_grad():
        temp = model(left).view(-1, 1, 300, 300)
    right.append(temp.to('cpu'))
    del temp
    torch.cuda.empty_cache()
```
Specifying no_grad() to my model tells PyTorch that I don't want to store any previous computations, thus freeing my GPU space.


Question: Evaluating pytorch models: `with torch.no_grad` vs `model.eval()` When I want to evaluate the performance of my model on the validation set, is it preferred to use with torch.no_grad: or model.eval()?

Response: TL;DR: Use both. They do different things, and have different scopes. with torch.no_grad - disables tracking of gradients in autograd. model.eval() changes the forward() behaviour of the module it is called upon eg, it disables dropout and has batch norm use the entire population statistics with torch.no_grad The torch.autograd.no_grad documentation says: Context-manager that disabled [sic] gradient calculation. Disabling gradient calculation is useful for inference, when you are sure that you will not call Tensor.backward(). It will reduce memory consumption for computations that would otherwise have requires_grad=True. In this mode, the result of every computation will have requires_grad=False, even when the inputs have requires_grad=True. model.eval() The nn.Module.eval documentation says: Sets the module in evaluation mode. This has any effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc. The creator of pytorch said the documentation should be updated to suggest the usage of both, and I raised the pull request.


Question:
PyTorch reshape tensor dimension I want to reshape a vector of shape (5,) into a matrix of shape (1, 5). With numpy, I can do:
```

```python
import numpy as np
```

```python
a = np.array([1, 2, 3, 4, 5])
```

```python
a.shape
#Output
#(5,)
```

```python
a = np.reshape(a, (1, 5))
```

```python
a.shape
#Output
#(1, 5)
```

```python
a
#Output
#array([[1, 2, 3, 4, 5]])
#```
#But how do I do this with PyTorch?
```

Response:
Use torch.unsqueeze(input, dim, out=None):
```

```python
import torch
```

```python
a = torch.Tensor([1, 2, 3, 4, 5])
```

```python
a
```

1
 2
 3
 4
 5
[torch.FloatTensor of size 5]

```python
a = a.unsqueeze(0)
```

```python
a
```

1  2  3  4  5
[torch.FloatTensor of size 1x5]
```


Question: What is the difference between torch.tensor and torch.Tensor? What is the difference between torch.tensor and torch.Tensor? What was the reasoning for providing these two very similar and confusing alternatives?

Response:
In PyTorch torch.Tensor is the main tensor class. So all tensors are just instances of torch.Tensor. When you call torch.Tensor() you will get an empty tensor without any data. In contrast torch.tensor is a function which returns a tensor. In the documentation it says:
```
torch.tensor(data, dtype=None, device=None, requires_grad=False) → Tensor
```
Constructs a tensor with data. This also explains why it is no problem creating an empty tensor instance of `torch.Tensor` without `data` by calling:
```
tensor_without_data = torch.Tensor()
```
But on the other side:
```
tensor_without_data = torch.tensor()
```
Will lead to an error:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-12-ebc3ceaa76d2> in <module>()
----> 1 torch.tensor()

TypeError: tensor() missing 1 required positional arguments: "data"
```
But in general there is no reason to choose `torch.Tensor` over `torch.tensor`. Also `torch.Tensor` lacks a docstring. Similar behaviour for creating a tensor without data like with: torch.Tensor() can be achieved using:
```
torch.tensor(())
```
Output:
```
tensor([])
```


Question:
`CrossEntropyLoss()` in PyTorch Cross entropy formula: But why does the following give loss = 0.7437 instead of loss = 0 (since 1*log(1) = 0)?
```python
import torch
import torch.nn as nn
from torch.autograd import Variable

output = Variable(torch.FloatTensor([0,0,0,1])).view(1, -1)
target = Variable(torch.LongTensor([3]))

criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
print(loss) # 0.7437
```

Response:
In your example you are treating output [0, 0, 0, 1] as probabilities as required by the mathematical definition of cross entropy. But PyTorch treats them as outputs, that don’t need to sum to 1, and need to be first converted into probabilities for which it uses the softmax function. So H(p, q) becomes:
```
H(p, softmax(output))
```
Translating the output [0, 0, 0, 1] into probabilities:
```
softmax([0, 0, 0, 1]) = [0.1749, 0.1749, 0.1749, 0.4754]
```
whence:
```
-log(0.4754) = 0.7437
```


Question: Pytorch: nn.Dropout vs. F.dropout There are two ways to perform dropout: torch.nn.Dropout torch.nn.functional.Dropout I ask: Is there a difference between them? When should I use one over the other? I don't see any performance difference when I switched them around.

Response:
The technical differences have already been shown in the other answer. However the main difference is that nn.Dropout is a torch Module itself which bears some convenience: A short example for illustration of some differences:
```py
import torch
import torch.nn as nn

class Model1(nn.Module):
    # Model 1 using functional dropout
    def __init__(self, p=0.0):
        super().__init__()
        self.p = p

    def forward(self, inputs):
        return nn.functional.dropout(inputs, p=self.p, training=True)

class Model2(nn.Module):
    # Model 2 using dropout module
    def __init__(self, p=0.0):
        super().__init__()
        self.drop_layer = nn.Dropout(p=p)

    def forward(self, inputs):
        return self.drop_layer(inputs)
model1 = Model1(p=0.5) # functional dropout
model2 = Model2(p=0.5) # dropout module

# creating inputs
inputs = torch.rand(10)
# forwarding inputs in train mode
print('Normal (train) model:')
print('Model 1', model1(inputs))
print('Model 2', model2(inputs))
print()

# switching to eval mode
model1.eval()
model2.eval()

# forwarding inputs in evaluation mode
print('Evaluation mode:')
print('Model 1', model1(inputs))
print('Model 2', model2(inputs))
# show model summary
print('Print summary:')
print(model1)
print(model2)
```
Output:
```py
Normal (train) model:
Model 1 tensor([ 1.5040,  0.0000,  0.0000,  0.8563,  0.0000,  0.0000,  1.5951,
         0.0000,  0.0000,  0.0946])
Model 2 tensor([ 0.0000,  0.3713,  1.9303,  0.0000,  0.0000,  0.3574,  0.0000,
         1.1273,  1.5818,  0.0946])

Evaluation mode:
Model 1 tensor([ 0.0000,  0.3713,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
         0.0000,  0.0000,  0.0000])
Model 2 tensor([ 0.7520,  0.1857,  0.9651,  0.4281,  0.7883,  0.1787,  0.7975,
         0.5636,  0.7909,  0.0473])
Print summary:
Model1()
Model2(
  (drop_layer): Dropout(p=0.5)
)
```
So which should I use? Both are completely equivalent in terms of applying dropout and even though the differences in usage are not that big, there are some reasons to favour the nn.Dropout over nn.functional.dropout: Dropout is designed to be only applied during training, so when doing predictions or evaluation of the model you want dropout to be turned off. The dropout module nn.Dropout conveniently handles this and shuts dropout off as soon as your model enters evaluation mode, while the functional dropout does not care about the evaluation / prediction mode. Even though you can set functional dropout to training=False to turn it off, it is still not such a convenient solution like with nn.Dropout. Also the drop rate is stored in the module, so you don't have to save it in an extra variable. In larger networks you might want to create different dropout layers with different drop rates - here nn.Dropout may increase readability and can bear also some convenience when using the layers multiple times. Finally, all modules which are assigned to your model are registered in your model. So you model class keeps track of them, that is why you can just turn off the dropout module by calling eval(). When using the functional dropout your model is not aware of it, thus it won't appear in any summary.


Question: Pytorch Operation to detect NaNs Is there a Pytorch-internal procedure to detect NaNs in Tensors? Tensorflow has the tf.is_nan and the tf.check_numerics operations ... Does Pytorch have something similar, somewhere? I could not find something like this in the docs... I am looking specifically for a Pytorch internal routine, since I would like this to happen on the GPU as well as on the CPU. This excludes numpy - based solutions (like np.isnan(sometensor.numpy()).any()) ...

Response:
You can always leverage the fact that nan != nan:
```

```python
x = torch.tensor([1, 2, np.nan])
#Output
#tensor([  1.,   2., nan.])
```

```python
x != x
#Output
#tensor([ 0,  0,  1], dtype=torch.uint8)
#```
#With pytorch 0.4 there is also torch.isnan:
#```
```

```python
torch.isnan(x)
#Output
#tensor([ 0,  0,  1], dtype=torch.uint8)
#```
```


Question: Differences in SciKit Learn, Keras, or Pytorch [closed] Closed. This question is opinion-based. It is not currently accepting answers. Want to improve this question? Because this question may lead to opinionated discussion, debate, and answers, it has been closed. You may edit the question if you feel you can improve it so that it requires answers that include facts and citations or a detailed explanation of the proposed solution. If edited, the question will be reviewed and might be reopened. Closed 6 years ago. The community reviewed whether to reopen this question 1 year ago and left it closed: Needs more focus Guide the asker to update the question so it focuses on a single, specific problem. Narrowing the question will help others answer the question concisely. You may edit the question if you feel you can improve it yourself. If edited, the question will be reviewed and might be reopened. Improve this question Are these libraries fairly interchangeable? Looking here, https://stackshare.io/stackups/keras-vs-pytorch-vs-scikit-learn, it seems the major difference is the underlying framework (at least for PyTorch).

Response: Yes, there is a major difference. SciKit Learn is a general machine learning library, built on top of NumPy. It features a lot of machine learning algorithms such as support vector machines, random forests, as well as a lot of utilities for general pre- and postprocessing of data. It is not a neural network framework. PyTorch is a deep learning framework, consisting of A vectorized math library similar to NumPy, but with GPU support and a lot of neural network related operations (such as softmax or various kinds of activations) Autograd - an algorithm which can automatically calculate gradients of your functions, defined in terms of the basic operations Gradient-based optimization routines for large scale optimization, dedicated to neural network optimization Neural-network related utility functions Keras is a higher-level deep learning framework, which abstracts many details away, making code simpler and more concise than in PyTorch or TensorFlow, at the cost of limited hackability. It abstracts away the computation backend, which can be TensorFlow, Theano or CNTK. It does not support a PyTorch backend, but that's not something unfathomable - you can consider it a simplified and streamlined subset of the above. In short, if you are going with "classic", non-neural algorithms, neither PyTorch nor Keras will be useful for you. If you're doing deep learning, scikit-learn may still be useful for its utility part; aside from it you will need the actual deep learning framework, where you can choose between Keras and PyTorch but you're unlikely to use both at the same time. This is very subjective, but in my view, if you're working on a novel algorithm, you're more likely to go with PyTorch (or TensorFlow or some other lower-level framework) for flexibility. If you're adapting a known and tested algorithm to a new problem setting, you may want to go with Keras for its greater simplicity and lower entry level.


Question: How to get a uniform distribution in a range [r1,r2] in PyTorch? I want to get a 2-D torch.Tensor with size [a,b] filled with values from a uniform distribution (in range [r1,r2]) in PyTorch.

Response:
If U is a random variable uniformly distributed on [0, 1], then (r1 - r2) * U + r2 is uniformly distributed on [r1, r2]. Thus, you just need:
```
(r1 - r2) * torch.rand(a, b) + r2
```
Alternatively, you can simply use:
```
torch.FloatTensor(a, b).uniform_(r1, r2)
```
To fully explain this formulation, let's look at some concrete numbers:
```py
r1 = 2 # Create uniform random numbers in half-open interval [2.0, 5.0)
r2 = 5

a = 1  # Create tensor shape 1 x 7
b = 7
```
We can break down the expression (r1 - r2) * torch.rand(a, b) + r2 as follows: torch.rand(a, b) produces an a x b (1x7) tensor with numbers uniformly distributed in the range [0.0, 1.0).
```py
x = torch.rand(a, b)
print(x)
# tensor([[0.5671, 0.9814, 0.8324, 0.0241, 0.2072, 0.6192, 0.4704]])
```
(r1 - r2) * torch.rand(a, b) produces numbers distributed in the uniform range [0.0, -3.0)
```py
print((r1 - r2) * x)
tensor([[-1.7014, -2.9441, -2.4972, -0.0722, -0.6216, -1.8577, -1.4112]])
```
(r1 - r2) * torch.rand(a, b) + r2 produces numbers in the uniform range [5.0, 2.0)
```py
print((r1 - r2) * x + r2)
tensor([[3.2986, 2.0559, 2.5028, 4.9278, 4.3784, 3.1423, 3.5888]])
```
Now, let's break down the answer suggested by @Jonasson: (r2 - r1) * torch.rand(a, b) + r1 Again, torch.rand(a, b) produces (1x7) numbers uniformly distributed in the range [0.0, 1.0).
```py
x = torch.rand(a, b)
print(x)
# tensor([[0.5671, 0.9814, 0.8324, 0.0241, 0.2072, 0.6192, 0.4704]])
```
(r2 - r1) * torch.rand(a, b) produces numbers uniformly distributed in the range [0.0, 3.0).
```py
print((r2 - r1) * x)
# tensor([[1.7014, 2.9441, 2.4972, 0.0722, 0.6216, 1.8577, 1.4112]])
```
(r2 - r1) * torch.rand(a, b) + r1 produces numbers uniformly distributed in the range [2.0, 5.0)
```py
print((r2 - r1) * x + r1)
tensor([[3.7014, 4.9441, 4.4972, 2.0722, 2.6216, 3.8577, 3.4112]])
```
In summary, (r1 - r2) * torch.rand(a, b) + r2 produces numbers in the range [r2, r1), while (r2 - r1) * torch.rand(a, b) + r1 produces numbers in the range [r1, r2).


Question: What are Torch Scripts in PyTorch? I've just found that PyTorch docs expose something that is called Torch Scripts. However, I do not know: When they should be used? How they should be used? What are their benefits?

Response: Torch Script is one of two modes of using the PyTorch just in time compiler, the other being tracing. The benefits are explained in the linked documentation: Torch Script is a way to create serializable and optimizable models from PyTorch code. Any code written in Torch Script can be saved from your Python process and loaded in a process where there is no Python dependency. The above quote is actually true both of scripting and tracing. So You gain the ability to serialize your models and later run them outside of Python, via LibTorch, a C++ native module. This allows you to embed your DL models in various production environments like mobile or IoT. There is an official guide on exporting models to C++ here. PyTorch can compile your jit-able modules rather than running them as an interpreter, allowing for various optimizations and improving performance, both during training and inference. This is equally helpful for development and production. Regarding Torch Script specifically, in comparison to tracing, it is a subset of Python, specified in detail here, which, when adhered to, can be compiled by PyTorch. It is more laborious to write Torch Script modules instead of tracing regular nn.Module subclasses, but it allows for some extra features over tracing, most notably flow control like if statements or for loops. Tracing treats such flow control as "constant" - in other words, if you have an if model.training clause in your module and trace it with training=True, it will always behave this way, even if you change the training variable to False later on. To answer your first question, you need to use jit if you want to deploy your models outside Python and otherwise you should use jit if you want to gain some execution performance at the price of extra development effort (as not every model can be straightforwardly made compliant with jit). In particular, you should use Torch Script if your code cannot be jited with tracing alone because it relies on some features such as if statements. For maximum ergonomy, you probably want to mix the two on a case-by-case basis. Finally, for how they should be used, please refer to all the documentation and tutorial links.


Question:
pytorch how to set .requires_grad False I want to set some of my model frozen. Following the official docs:
```python
with torch.no_grad():
    linear = nn.Linear(1, 1)
    linear.eval()
    print(linear.weight.requires_grad)
```
But it prints True instead of False. If I want to set the model in eval mode, what should I do?

Response:
requires_grad=False If you want to freeze part of your model and train the rest, you can set requires_grad of the parameters you want to freeze to False. For example, if you only want to keep the convolutional part of VGG16 fixed:
```
model = torchvision.models.vgg16(pretrained=True)
for param in model.features.parameters():
    param.requires_grad = False
```
By switching the requires_grad flags to False, no intermediate buffers will be saved, until the computation gets to some point where one of the inputs of the operation requires the gradient. torch.no_grad() Using the context manager torch.no_grad is a different way to achieve that goal: in the no_grad context, all the results of the computations will have requires_grad=False, even if the inputs have requires_grad=True. Notice that you won't be able to backpropagate the gradient to layers before the no_grad. For example:
```
x = torch.randn(2, 2)
x.requires_grad = True

lin0 = nn.Linear(2, 2)
lin1 = nn.Linear(2, 2)
lin2 = nn.Linear(2, 2)
x1 = lin0(x)
with torch.no_grad():
    x2 = lin1(x1)
x3 = lin2(x2)
x3.sum().backward()
print(lin0.weight.grad, lin1.weight.grad, lin2.weight.grad)
```
outputs:
```
(None, None, tensor([[-1.4481, -1.1789],
         [-1.4481, -1.1789]]))
```
Here lin1.weight.requires_grad was True, but the gradient wasn't computed because the oepration was done in the no_grad context. model.eval() If your goal is not to finetune, but to set your model in inference mode, the most convenient way is to use the torch.no_grad context manager. In this case you also have to set your model to evaluation mode, this is achieved by calling eval() on the nn.Module, for example:
```
model = torchvision.models.vgg16(pretrained=True)
model.eval()
```
This operation sets the attribute self.training of the layers to False, in practice this will change the behavior of operations like Dropout or BatchNorm that must behave differently at training and test time.


Question: PyTorch: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead Calling tensor.numpy() gives the error: RuntimeError: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead. tensor.cpu().detach().numpy() gives the same error.

Response:
Error reproduced
```
import torch

tensor1 = torch.tensor([1.0,2.0],requires_grad=True)

print(tensor1)
print(type(tensor1))

tensor1 = tensor1.numpy()

print(tensor1)
print(type(tensor1))
```
which leads to the exact same error for the line tensor1 = tensor1.numpy():
```
tensor([1., 2.], requires_grad=True)
<class 'torch.Tensor'>
Traceback (most recent call last):
  File "/home/badScript.py", line 8, in <module>
    tensor1 = tensor1.numpy()
RuntimeError: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead.

Process finished with exit code 1
```
Generic solution this was suggested to you in your error message, just replace var with your variable name
```
import torch

tensor1 = torch.tensor([1.0,2.0],requires_grad=True)

print(tensor1)
print(type(tensor1))

tensor1 = tensor1.detach().numpy()

print(tensor1)
print(type(tensor1))
```
which returns as expected
```
tensor([1., 2.], requires_grad=True)
<class 'torch.Tensor'>
[1. 2.]
<class 'numpy.ndarray'>

Process finished with exit code 0
```
Some explanation You need to convert your tensor to another tensor that isn't requiring a gradient in addition to its actual value definition. This other tensor can be converted to a numpy array. Cf. this discuss.pytorch post. (I think, more precisely, that one needs to do that in order to get the actual tensor out of its pytorch Variable wrapper, cf. this other discuss.pytorch post).


Question:
PyTorch - RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed I keep running into this error: RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time. I had searched in Pytorch forum, but still can’t find out what I have done wrong in my custom loss function. My model is nn.GRU, and here is my custom loss function:
```
def _loss(outputs, session, items):  # `items` is a dict() contains embedding of all items
    def f(output, target):
        pos = torch.from_numpy(np.array([items[target["click"]]])).float()
        neg = torch.from_numpy(np.array([items[idx] for idx in target["suggest_list"] if idx != target["click"]])).float()
        if USE_CUDA:
            pos, neg = pos.cuda(), neg.cuda()
        pos, neg = Variable(pos), Variable(neg)

        pos = F.cosine_similarity(output, pos)
        if neg.size()[0] == 0:
            return torch.mean(F.logsigmoid(pos))
        neg = F.cosine_similarity(output.expand_as(neg), neg)

        return torch.mean(F.logsigmoid(pos - neg))

    loss = map(f, outputs, session)
    return -torch.mean(torch.cat(loss))
```
Training code:
```
# zero the parameter gradients
    model.zero_grad()

    # forward + backward + optimize
    outputs, hidden = model(inputs, hidden)
    loss = _loss(outputs, session, items)
    acc_loss += loss.data[0]

    loss.backward()
    # Add parameters' gradients to their values, multiplied by learning rate
    for p in model.parameters():
        p.data.add_(-learning_rate, p.grad.data)
```

Response: The problem is from my training loop: it doesn’t detach or repackage the hidden state in between batches? If so, then loss.backward() is trying to back-propagate all the way through to the start of time, which works for the first batch but not for the second because the graph for the first batch has been discarded. there are two possible solutions. detach/repackage the hidden state in between batches. There are (at least) three ways to do this (and I chose this solution): hidden.detach_() (or equivalently hidden = hidden.detach()). replace loss.backward() with loss.backward(retain_graph=True) but know that each successive batch will take more time than the previous one because it will have to back-propagate all the way through to the start of the first batch. Example


Question: Is .data still useful in pytorch? I'm new to pytorch. I read much pytorch code which heavily uses tensor's .data member. But I search .data in the official document and Google, finding little. I guess .data contains the data in the tensor, but I don't know when we need it and when not?

Response: .data was an attribute of Variable (object representing Tensor with history tracking e.g. for automatic update), not Tensor. Actually, .data was giving access to the Variable's underlying Tensor. However, since PyTorch version 0.4.0, Variable and Tensor have been merged (into an updated Tensor structure), so .data disappeared along the previous Variable object (well Variable is still there for backward-compatibility, but is deprecated). Paragraph from Release Notes for version 0.4.0 (I recommend reading the whole section about Variable/Tensor updates): What about .data? .data was the primary way to get the underlying Tensor from a Variable. After this merge, calling y = x.data still has similar semantics. So y will be a Tensor that shares the same data with x, is unrelated with the computation history of x, and has requires_grad=False. However, .data can be unsafe in some cases. Any changes on x.data wouldn't be tracked by autograd, and the computed gradients would be incorrect if x is needed in a backward pass. A safer alternative is to use x.detach(), which also returns a Tensor that shares data with requires_grad=False, but will have its in-place changes reported by autograd if x is needed in backward.


Question: Understanding PyTorch einsum I'm familiar with how einsum works in NumPy. A similar functionality is also offered by PyTorch: torch.einsum(). What are the similarities and differences, either in terms of functionality or performance? The information available at PyTorch documentation is rather scanty and doesn't provide any insights regarding this.

Response:
Since the description of einsum is skimpy in torch documentation, I decided to write this post to document, compare and contrast how torch.einsum() behaves when compared to numpy.einsum(). Differences: NumPy allows both small case and capitalized letters [a-zA-Z] for the "subscript string" whereas PyTorch allows only the small case letters [a-z]. NumPy accepts nd-arrays, plain Python lists (or tuples), list of lists (or tuple of tuples, list of tuples, tuple of lists) or even PyTorch tensors as operands (i.e. inputs). This is because the operands have only to be array_like and not strictly NumPy nd-arrays. On the contrary, PyTorch expects the operands (i.e. inputs) strictly to be PyTorch tensors. It will throw a TypeError if you pass either plain Python lists/tuples (or its combinations) or NumPy nd-arrays. NumPy supports lot of keyword arguments (for e.g. optimize) in addition to nd-arrays while PyTorch doesn't offer such flexibility yet. Here are the implementations of some examples both in PyTorch and NumPy:
```
# input tensors to work with

```python
vec
#Output
#tensor([0, 1, 2, 3])
```

```python
aten
#Output
#tensor([[11, 12, 13, 14],
#        [21, 22, 23, 24],
#        [31, 32, 33, 34],
#        [41, 42, 43, 44]])
```

```python
bten
#Output
#tensor([[1, 1, 1, 1],
#        [2, 2, 2, 2],
#        [3, 3, 3, 3],
#        [4, 4, 4, 4]])
#```
#1) Matrix multiplication PyTorch: torch.matmul(aten, bten) ; aten.mm(bten) NumPy : np.einsum("ij, jk -> ik", arr1, arr2)
#```
```

```python
torch.einsum('ij, jk -> ik', aten, bten)
#Output
#tensor([[130, 130, 130, 130],
#        [230, 230, 230, 230],
#        [330, 330, 330, 330],
#        [430, 430, 430, 430]])
#```
#2) Extract elements along the main-diagonal PyTorch: torch.diag(aten) NumPy : np.einsum("ii -> i", arr)
#```
```

```python
torch.einsum('ii -> i', aten)
#Output
#tensor([11, 22, 33, 44])
#```
#3) Hadamard product (i.e. element-wise product of two tensors) PyTorch: aten * bten NumPy : np.einsum("ij, ij -> ij", arr1, arr2)
#```
```

```python
torch.einsum('ij, ij -> ij', aten, bten)
#Output
#tensor([[ 11,  12,  13,  14],
#        [ 42,  44,  46,  48],
#        [ 93,  96,  99, 102],
#        [164, 168, 172, 176]])
#```
#4) Element-wise squaring PyTorch: aten ** 2 NumPy : np.einsum("ij, ij -> ij", arr, arr)
#```
```

```python
torch.einsum('ij, ij -> ij', aten, aten)
#Output
#tensor([[ 121,  144,  169,  196],
#        [ 441,  484,  529,  576],
#        [ 961, 1024, 1089, 1156],
#        [1681, 1764, 1849, 1936]])
#```
#General: Element-wise nth power can be implemented by repeating the subscript string and tensor n times. For e.g., computing element-wise 4th power of a tensor can be done using:
#```
## NumPy: np.einsum('ij, ij, ij, ij -> ij', arr, arr, arr, arr)
```

```python
torch.einsum('ij, ij, ij, ij -> ij', aten, aten, aten, aten)
#Output
#tensor([[  14641,   20736,   28561,   38416],
#        [ 194481,  234256,  279841,  331776],
#        [ 923521, 1048576, 1185921, 1336336],
#        [2825761, 3111696, 3418801, 3748096]])
#```
#5) Trace (i.e. sum of main-diagonal elements) PyTorch: torch.trace(aten) NumPy einsum: np.einsum("ii -> ", arr)
#```
```

```python
torch.einsum('ii -> ', aten)
#Output
#tensor(110)
#```
#6) Matrix transpose PyTorch: torch.transpose(aten, 1, 0) NumPy einsum: np.einsum("ij -> ji", arr)
#```
```

```python
torch.einsum('ij -> ji', aten)
#Output
#tensor([[11, 21, 31, 41],
#        [12, 22, 32, 42],
#        [13, 23, 33, 43],
#        [14, 24, 34, 44]])
#```
#7) Outer Product (of vectors) PyTorch: torch.ger(vec, vec) NumPy einsum: np.einsum("i, j -> ij", vec, vec)
#```
```

```python
torch.einsum('i, j -> ij', vec, vec)
#Output
#tensor([[0, 0, 0, 0],
#        [0, 1, 2, 3],
#        [0, 2, 4, 6],
#        [0, 3, 6, 9]])
#```
#8) Inner Product (of vectors) PyTorch: torch.dot(vec1, vec2) NumPy einsum: np.einsum("i, i -> ", vec1, vec2)
#```
```

```python
torch.einsum('i, i -> ', vec, vec)
#Output
#tensor(14)
#```
#9) Sum along axis 0 PyTorch: torch.sum(aten, 0) NumPy einsum: np.einsum("ij -> j", arr)
#```
```

```python
torch.einsum('ij -> j', aten)
#Output
#tensor([104, 108, 112, 116])
#```
#10) Sum along axis 1 PyTorch: torch.sum(aten, 1) NumPy einsum: np.einsum("ij -> i", arr)
#```
```

```python
torch.einsum('ij -> i', aten)
#Output
#tensor([ 50,  90, 130, 170])
#```
#11) Batch Matrix Multiplication PyTorch: torch.bmm(batch_tensor_1, batch_tensor_2) NumPy : np.einsum("bij, bjk -> bik", batch_tensor_1, batch_tensor_2)
#```
## input batch tensors to work with
```

```python
batch_tensor_1 = torch.arange(2 * 4 * 3).reshape(2, 4, 3)
```

```python
batch_tensor_2 = torch.arange(2 * 3 * 4).reshape(2, 3, 4)
```

```python
torch.bmm(batch_tensor_1, batch_tensor_2)
#Output
#tensor([[[  20,   23,   26,   29],
#         [  56,   68,   80,   92],
#         [  92,  113,  134,  155],
#         [ 128,  158,  188,  218]],
#
#        [[ 632,  671,  710,  749],
#         [ 776,  824,  872,  920],
#         [ 920,  977, 1034, 1091],
#         [1064, 1130, 1196, 1262]]])
#
## sanity check with the shapes
```

```python
torch.bmm(batch_tensor_1, batch_tensor_2).shape
#Output
#torch.Size([2, 4, 4])
#
## batch matrix multiply using einsum
```

```python
torch.einsum("bij, bjk -> bik", batch_tensor_1, batch_tensor_2)
#Output
#tensor([[[  20,   23,   26,   29],
#         [  56,   68,   80,   92],
#         [  92,  113,  134,  155],
#         [ 128,  158,  188,  218]],
#
#        [[ 632,  671,  710,  749],
#         [ 776,  824,  872,  920],
#         [ 920,  977, 1034, 1091],
#         [1064, 1130, 1196, 1262]]])
#
## sanity check with the shapes
```

```python
torch.einsum("bij, bjk -> bik", batch_tensor_1, batch_tensor_2).shape
```
12) Sum along axis 2 PyTorch: torch.sum(batch_ten, 2) NumPy einsum: np.einsum("ijk -> ij", arr3D)
```
```

```python
torch.einsum("ijk -> ij", batch_ten)
#Output
#tensor([[ 50,  90, 130, 170],
#        [  4,   8,  12,  16]])
#```
#13) Sum all the elements in an nD tensor PyTorch: torch.sum(batch_ten) NumPy einsum: np.einsum("ijk -> ", arr3D)
#```
```

```python
torch.einsum("ijk -> ", batch_ten)
#Output
#tensor(480)
#```
#14) Sum over multiple axes (i.e. marginalization) PyTorch: torch.sum(arr, dim=(dim0, dim1, dim2, dim3, dim4, dim6, dim7)) NumPy: np.einsum("ijklmnop -> n", nDarr)
#```
## 8D tensor
```

```python
nDten = torch.randn((3,5,4,6,8,2,7,9))
```

```python
nDten.shape
#Output
#torch.Size([3, 5, 4, 6, 8, 2, 7, 9])
#
## marginalize out dimension 5 (i.e. "n" here)
```

```python
esum = torch.einsum("ijklmnop -> n", nDten)
```

```python
esum
#Output
#tensor([  98.6921, -206.0575])
#
## marginalize out axis 5 (i.e. sum over rest of the axes)
```

```python
tsum = torch.sum(nDten, dim=(0, 1, 2, 3, 4, 6, 7))
```

```python
torch.allclose(tsum, esum)
#Output
#True
#```
#15) Double Dot Products / Frobenius inner product (same as: torch.sum(hadamard-product) cf. 3) PyTorch: torch.sum(aten * bten) NumPy : np.einsum("ij, ij -> ", arr1, arr2)
#```
```

```python
torch.einsum("ij, ij -> ", aten, bten)
#Output
#tensor(1300)
#```
```


Question: reshaping a tensor with padding in pytorch How do I reshape a tensor with dimensions (30, 35, 49) to (30, 35, 512) by padding it?

Response:
While @nemo's solution works fine, there is a pytorch internal routine, torch.nn.functional.pad, that does the same - and which has a couple of properties that a torch.ones(*sizes)*pad_value solution does not (namely other forms of padding, like reflection padding or replicate padding ... it also checks some gradient-related properties):
```py
import torch.nn.functional as F
source = torch.rand((5,10))
# now we expand to size (7, 11) by appending a row of 0s at pos 0 and pos 6,
# and a column of 0s at pos 10
result = F.pad(input=source, pad=(0, 1, 1, 1), mode='constant', value=0)
```
The semantics of the arguments are: input: the source tensor, pad: a list of length 2 * len(source.shape) of the form (begin last axis, end last axis, begin 2nd to last axis, end 2nd to last axis, begin 3rd to last axis, etc.) that states how many dimensions should be added to the beginning and end of each axis, mode: 'constant', 'reflect' or 'replicate'. Default: 'constant' for the different kinds of padding value for constant padding.


Question:
How to remove the last FC layer from a ResNet model in PyTorch? I am using a ResNet152 model from PyTorch. I'd like to strip off the last FC layer from the model. Here's my code:
```
from torchvision import datasets, transforms, models
model = models.resnet152(pretrained=True)
print(model)
```
When I print the model, the last few lines look like this:
```
(2):  Bottleneck(
      (conv1):  Conv2d(2048,  512,  kernel_size=(1,  1),  stride=(1,  1),  bias=False)
      (bn1):  BatchNorm2d(512,  eps=1e-05,  momentum=0.1,  affine=True,  track_running_stats=True)
      (conv2):  Conv2d(512,  512,  kernel_size=(3,  3),  stride=(1,  1),  padding=(1,  1),  bias=False)
      (bn2):  BatchNorm2d(512,  eps=1e-05,  momentum=0.1,  affine=True,  track_running_stats=True)
      (conv3):  Conv2d(512,  2048,  kernel_size=(1,  1),  stride=(1,  1),  bias=False)
      (bn3):  BatchNorm2d(2048,  eps=1e-05,  momentum=0.1,  affine=True,  track_running_stats=True)
      (relu):  ReLU(inplace)
    )
  )
  (avgpool):  AvgPool2d(kernel_size=7,  stride=1,  padding=0)
  (fc):  Linear(in_features=2048,  out_features=1000,  bias=True)
)
```
I want to remove that last fc layer from the model. I found an answer here on SO (How to convert pretrained FC layers to CONV layers in Pytorch), where mexmex seems to provide the answer I'm looking for:
```
list(model.modules()) # to inspect the modules of your model
my_model = nn.Sequential(*list(model.modules())[:-1]) # strips off last linear layer
```
So I added those lines to my code like this:
```
model = models.resnet152(pretrained=True)
list(model.modules()) # to inspect the modules of your model
my_model = nn.Sequential(*list(model.modules())[:-1]) # strips off last linear layer
print(my_model)
```
But this code doesn't work as advertised -- as least not for me. The rest of this post is a detailed explanation of why that answer doesn't work so this question doesn't get closed as a duplicate. First, the printed model is nearly 5x larger than before. I see the same model as before, but followed by what appears to be a repeat of the model, but perhaps flattened.
```
(2): Bottleneck(
      (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
    )
  )
  (avgpool): AvgPool2d(kernel_size=7, stride=1, padding=0)
  (fc): Linear(in_features=2048, out_features=1000, bias=True)
)
(1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
(2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(3): ReLU(inplace)
(4): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
(5): Sequential(
  . . . this goes on for ~1600 more lines . . .
  (415): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (416): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (417): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (418): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (419): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (420): ReLU(inplace)
  (421): AvgPool2d(kernel_size=7, stride=1, padding=0)
)
```
Second, the fc layer is still there -- and the Conv2D layer after it looks just like the first layer of ResNet152. Third, if I try to invoke my_model.forward(), pytorch complains about a size mismatch. It expects size [1, 3, 224, 224], but the input was [1, 1000]. So it looks like a copy of the entire model (minus the fc layer) is getting appended to the original model. Bottom line, the only answer I found on SO doesn't actually work.

Response:
For ResNet model, you can use children attribute to access layers since ResNet model in pytorch consist of nn modules. (Tested on pytorch 0.4.1)
```
model = models.resnet152(pretrained=True)
newmodel = torch.nn.Sequential(*(list(model.children())[:-1]))
print(newmodel)
```
Update: Although there is not an universal answer for the question that can work on all pytorch models, it should work on all well structured ones. Existing layers you add to your model (such as torch.nn.Linear, torch.nn.Conv2d, torch.nn.BatchNorm2d...) all based on torch.nn.Module class. And if you implement a custom layer and add that to your network you should inherit it from pytorch's torch.nn.Module class. As written in documentation, children attribute lets you access the modules of your class/model/network.
```
def children(self):
        r"""Returns an iterator over immediate children modules.
```
Update: It is important to note that children() returns "immediate" modules, which means if last module of your network is a sequential, it will return whole sequential.


Question: How to repeat tensor in a specific new dimension in PyTorch If I have a tensor A which has shape [M, N], I want to repeat the tensor K times so that the result B has shape [M, K, N] and each slice B[:, k, :] should has the same data as A. Which is the best practice without a for loop. K might be in other dimension. torch.repeat_interleave() and tensor.repeat() does not seem to work. Or I am using it in a wrong way.

Response:
tensor.repeat should suit your needs but you need to insert a unitary dimension first. For this we could use either tensor.unsqueeze or tensor.reshape. Since unsqueeze is specifically defined to insert a unitary dimension we will use that.
```
B = A.unsqueeze(1).repeat(1, K, 1)
```
Code Description A.unsqueeze(1) turns A from an [M, N] to [M, 1, N] and .repeat(1, K, 1) repeats the tensor K times along the second dimension.


Question:
Should I use softmax as output when using cross entropy loss in PyTorch? I have a problem with classifying fully connected deep neural net with 2 hidden layers for MNIST dataset in pytorch. I want to use tanh as activations in both hidden layers, but in the end, I should use softmax. For the loss, I am choosing nn.CrossEntropyLoss() in PyTorch, which (as I have found out) does not want to take one-hot encoded labels as true labels, but takes LongTensor of classes instead. My model is nn.Sequential() and when I am using softmax in the end, it gives me worse results in terms of accuracy on testing data. Why?
```py
import torch
from torch import nn

inputs, n_hidden0, n_hidden1, out = 784, 128, 64, 10
n_epochs = 500
model = nn.Sequential(
    nn.Linear(inputs, n_hidden0, bias=True),
    nn.Tanh(),
    nn.Linear(n_hidden0, n_hidden1, bias=True),
    nn.Tanh(),
    nn.Linear(n_hidden1, out, bias=True),
    nn.Softmax()  # SHOULD THIS BE THERE?
)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.5)

for epoch in range(n_epochs):
    y_pred = model(X_train)
    loss = criterion(y_pred, Y_train)
    print('epoch: ', epoch+1,' loss: ', loss.item())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
```

Response: As stated in the torch.nn.CrossEntropyLoss() doc: This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class. Therefore, you should not use softmax before.


Question: In distributed computing, what are world size and rank? I've been reading through some documentation and example code with the end goal of writing scripts for distributed computing (running PyTorch), but the concepts confuse me. Let's assume that we have a single node with 4 GPUs, and we want to run our script on those 4 GPUs (i.e. one process per GPU). In such a scenario, what are the rank world size and rank? I often find the explanation for world size: Total number of processes involved in the job, so I assume that that is four in our example, but what about rank? To explain it further, another example with multiple nodes and multiple GPUs could be useful, too.

Response: These concepts are related to parallel computing. It would be helpful to learn a little about parallel computing, e.g., MPI. You can think of world as a group containing all the processes for your distributed training. Usually, each GPU corresponds to one process. Processes in the world can communicate with each other, which is why you can train your model distributedly and still get the correct gradient update. So world size is the number of processes for your training, which is usually the number of GPUs you are using for distributed training. Rank is the unique ID given to a process, so that other processes know how to identify a particular process. Local rank is the a unique local ID for processes running in a single node, this is where my view differs with @zihaozhihao. Let's take a concrete example. Suppose we run our training in 2 servers (some articles also call them nodes) and each server/node has 4 GPUs. The world size is 4*2=8. The ranks for the processes will be [0, 1, 2, 3, 4, 5, 6, 7]. In each node, the local rank will be [0, 1, 2, 3]. I have also written a post about MPI collectives and basic concepts. The link is here.


Question:
How to use 'collate_fn' with dataloaders? I am trying to train a pretrained roberta model using 3 inputs, 3 input_masks and a label as tensors of my training dataset. I do this using the following code:
```
from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler
batch_size = 32
# Create the DataLoader for our training set.
train_data = TensorDataset(train_AT, train_BT, train_CT, train_maskAT, train_maskBT, train_maskCT, labels_trainT)
train_dataloader = DataLoader(train_data, batch_size=batch_size)

# Create the Dataloader for our validation set.
validation_data = TensorDataset(val_AT, val_BT, val_CT, val_maskAT, val_maskBT, val_maskCT, labels_valT)
val_dataloader = DataLoader(validation_data, batch_size=batch_size)

# Pytorch Training
training_args = TrainingArguments(
    output_dir='C:/Users/samvd/Documents/Master/AppliedMachineLearning/FinalProject/results',          # output directory
    num_train_epochs=1,              # total # of training epochs
    per_device_train_batch_size=32,  # batch size per device during training
    per_device_eval_batch_size=32,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='C:/Users/samvd/Documents/Master/AppliedMachineLearning/FinalProject/logs',            # directory for storing logs
)

trainer = Trainer(
    model=model,                          # the instantiated 🤗 Transformers model to be trained
    args=training_args,                   # training arguments, defined above
    train_dataset = train_data,           # training dataset
    eval_dataset = validation_data,       # evaluation dataset
)

trainer.train()
```
However this gives me the following error: TypeError: vars() argument must have dict attribute Now I have found out that it is probably because I don't use collate_fn when using DataLoader, but I can't really find a source that helps me define this correctly so the trainer understands the different tensors I put in. Can anyone point me in the right direction?

Response:
Basically, the collate_fn receives a list of tuples if your __getitem__ function from a Dataset subclass returns a tuple, or just a normal list if your Dataset subclass returns only one element. Its main objective is to create your batch without spending much time implementing it manually. Try to see it as a glue that you specify the way examples stick together in a batch. If you don’t use it, PyTorch only put batch_size examples together as you would using torch.stack (not exactly it, but it is simple like that). Suppose for example, you want to create batches of a list of varying dimension tensors. The below code pads sequences with 0 until the maximum sequence size of the batch, that is why we need the collate_fn, because a standard batching algorithm (simply using torch.stack) won’t work in this case, and we need to manually pad different sequences with variable length to the same size before creating the batch.
```
def collate_fn(data):
    """
       data: is a list of tuples with (example, label, length)
             where 'example' is a tensor of arbitrary shape
             and label/length are scalars
    """
    _, labels, lengths = zip(*data)
    max_len = max(lengths)
    n_ftrs = data[0][0].size(1)
    features = torch.zeros((len(data), max_len, n_ftrs))
    labels = torch.tensor(labels)
    lengths = torch.tensor(lengths)

    for i in range(len(data)):
        j, k = data[i][0].size(0), data[i][0].size(1)
        features[i] = torch.cat([data[i][0], torch.zeros((max_len - j, k))])

    return features.float(), labels.long(), lengths.long()
```
The function above is fed to the collate_fn param in the DataLoader, as this example:
```
DataLoader(toy_dataset, collate_fn=collate_fn, batch_size=5)
```
With this collate_fn function, you always gonna have a tensor where all your examples have the same size. So, when you feed your forward() function with this data, you need to use the length to get the original data back, to not use those meaningless zeros in your computation. Source: Pytorch Forum


Question:
Cannot convert list to array: ValueError: only one element tensors can be converted to Python scalars I'm currently working with the PyTorch framework and trying to understand foreign code. I got an indices issue and wanted to print the shape of a list. The only way of doing so (as far as Google tells me) is to convert the list into a numpy array and then getting the shape with numpy.ndarray.shape(). But trying to convert my list into an array, I got a ValueError: only one element tensors can be converted to Python scalars. My List is a converted PyTorch Tensor (list(pytorchTensor)) and looks somewhat like this:
```
[
tensor([[-0.2781, -0.2567, -0.2353,  ..., -0.9640, -0.9855, -1.0069],
        [-0.2781, -0.2567, -0.2353,  ..., -1.0069, -1.0283, -1.0927],
        [-0.2567, -0.2567, -0.2138,  ..., -1.0712, -1.1141, -1.1784],
        ...,
        [-0.6640, -0.6425, -0.6211,  ..., -1.0712, -1.1141, -1.0927],
        [-0.6640, -0.6425, -0.5997,  ..., -0.9426, -0.9640, -0.9640],
        [-0.6640, -0.6425, -0.5997,  ..., -0.9640, -0.9426, -0.9426]]),

tensor([[-0.0769, -0.0980, -0.0769,  ..., -0.9388, -0.9598, -0.9808],
        [-0.0559, -0.0769, -0.0980,  ..., -0.9598, -1.0018, -1.0228],
        [-0.0559, -0.0769, -0.0769,  ..., -1.0228, -1.0439, -1.0859],
        ...,
        [-0.4973, -0.4973, -0.4973,  ..., -1.0018, -1.0439, -1.0228],
        [-0.4973, -0.4973, -0.4973,  ..., -0.8757, -0.9177, -0.9177],
        [-0.4973, -0.4973, -0.4973,  ..., -0.9177, -0.8967, -0.8967]]),

tensor([[-0.1313, -0.1313, -0.1100,  ..., -0.8115, -0.8328, -0.8753],
        [-0.1313, -0.1525, -0.1313,  ..., -0.8541, -0.8966, -0.9391],
        [-0.1100, -0.1313, -0.1100,  ..., -0.9391, -0.9816, -1.0666],
        ...,
        [-0.4502, -0.4714, -0.4502,  ..., -0.8966, -0.8966, -0.8966],
        [-0.4502, -0.4714, -0.4502,  ..., -0.8115, -0.8115, -0.7903],
        [-0.4502, -0.4714, -0.4502,  ..., -0.8115, -0.7690, -0.7690]]),
]
```
Is there a way of getting the shape of that list without converting it into a numpy array?

Response:
It seems like you have a list of tensors. For each tensor you can see its size() (no need to convert to list/numpy). If you insist, you can convert a tensor to numpy array using numpy(): Return a list of tensor shapes:
```
>> [t.size() for t in my_list_of_tensors]
```
Returns a list of numpy arrays:
```
>> [t.numpy() for t in my_list_of_tensors]
```
In terms of performance, it is always best to avoid casting of tensors into numpy arrays, as it may incur sync of device/host memory. If you only need to check the shape of a tensor, use size() function.


Question:
How to cast a 1-d IntTensor to int in Pytorch How do I convert a 1-D IntTensor to an integer? This:
```
IntTensor.int()
```
Gives an error:
```
KeyError: Variable containing:
 423
[torch.IntTensor of size 1]
```

Response:
The simplest and cleanest method I know:
```
IntTensor.item()
```
Returns the value of this tensor as a standard Python number. This only works for tensors with one element. For other cases, see tolist. PyTorch Docs