Question:
Understanding torch.nn.LayerNorm in nlp I'm trying to understanding how torch.nn.LayerNorm works in a nlp model. Asuming the input data is a batch of sequence of word embeddings:
```
batch_size, seq_size, dim = 2, 3, 4
embedding = torch.randn(batch_size, seq_size, dim)
print("x: ", embedding)

layer_norm = torch.nn.LayerNorm(dim)
print("y: ", layer_norm(embedding))

# outputs:
"""
x:  tensor([[[ 0.5909,  0.1326,  0.8100,  0.7631],
         [ 0.5831, -1.7923, -0.1453, -0.6882],
         [ 1.1280,  1.6121, -1.2383,  0.2150]],

        [[-0.2128, -0.5246, -0.0511,  0.2798],
         [ 0.8254,  1.2262, -0.0252, -1.9972],
         [-0.6092, -0.4709, -0.8038, -1.2711]]])
y:  tensor([[[ 0.0626, -1.6495,  0.8810,  0.7060],
         [ 1.2621, -1.4789,  0.4216, -0.2048],
         [ 0.6437,  1.0897, -1.5360, -0.1973]],

        [[-0.2950, -1.3698,  0.2621,  1.4027],
         [ 0.6585,  0.9811, -0.0262, -1.6134],
         [ 0.5934,  1.0505, -0.0497, -1.5942]]],
       grad_fn=<NativeLayerNormBackward0>)
"""
```
From the document's description, my understanding is that the mean and std are computed by all embedding values per sample. So I try to compute y[0, 0, :] manually:
```
mean = torch.mean(embedding[0, :, :])
std = torch.std(embedding[0, :, :])
print((embedding[0, 0, :] - mean) / std)
```
which gives tensor([ 0.4310, -0.0319, 0.6523, 0.6050]) and that's not the right output. I want to know what is the right way to compute y[0, 0, :]?

Response:
Pytorch layer norm states mean and std calculated over last D dimensions. Based on this as I expect for (batch_size, seq_size, embedding_dim) here calculation should be over (seq_size, embedding_dim) for layer norm as last 2 dimensions excluding batch dim. A similar question and answer with layer norm implementation can be found here, layer Normalization in pytorch?. In some paper below it shows different layer norm application in NLP. Explanation of Intance vs Layer vs Group Norm From group norm paper Layer Normalization (LN) operates along the channel dimension LN computes µ and σ along the (C, H, W) axes for each sample. Different Application Example In pytorch doc for NLP 3d tensor example mean and std instead are calculated over only last dim embedding_dim. In this paper it shows similar to pytorch doc example, almost all NLP tasks take variable length sequences as input, which is very suitable for LN that only calculates statistics in the channel dimension without involving the batch and sequence length dimension. Example shown in Another paper, LN normalizes across the channel/feature dimension as shown in Figure 1. Manual Layer Norm with only Embed Dim
```
import torch

batch_size, seq_size, dim = 2, 3, 4
last_dims = 4

embedding = torch.randn(batch_size, seq_size, dim)
print("x: ", embedding)

layer_norm = torch.nn.LayerNorm(last_dims, elementwise_affine = False)
layer_norm_out = layer_norm(embedding)
print("y: ", layer_norm_out)

eps: float = 0.00001
mean = torch.mean(embedding[0, :, :], dim=(-1), keepdim=True)
var = torch.square(embedding[0, :, :] - mean).mean(dim=(-1), keepdim=True)
y_custom = (embedding[0, :, :] - mean) / torch.sqrt(var + eps)
print("y_custom: ", y_custom)
assert torch.allclose(layer_norm_out[0], y_custom), 'Tensors do not match.'

eps: float = 0.00001
mean = torch.mean(embedding[1, :, :], dim=(-1), keepdim=True)
var = torch.square(embedding[1, :, :] - mean).mean(dim=(-1), keepdim=True)
y_custom = (embedding[1, :, :] - mean) / torch.sqrt(var + eps)
print("y_custom: ", y_custom)
assert torch.allclose(layer_norm_out[1], y_custom), 'Tensors do not match.'
```
Output
```
x:  tensor([[[-0.0594, -0.8702, -1.9837,  0.2914],
         [-0.4774,  1.0372,  0.6425, -1.1357],
         [ 0.3872, -0.9190, -0.5774,  0.3281]],

        [[-0.5548,  0.0815,  0.2333,  0.3569],
         [ 1.0380, -0.1756, -0.7417,  2.2930],
         [-0.0075, -0.3623,  1.9310, -0.7043]]])
y:  tensor([[[ 0.6813, -0.2454, -1.5180,  1.0822],
         [-0.5700,  1.1774,  0.7220, -1.3295],
         [ 1.0285, -1.2779, -0.6747,  0.9241]],

        [[-1.6638,  0.1490,  0.5814,  0.9334],
         [ 0.3720, -0.6668, -1.1513,  1.4462],
         [-0.2171, -0.5644,  1.6809, -0.8994]]])
y_custom:  tensor([[ 0.6813, -0.2454, -1.5180,  1.0822],
        [-0.5700,  1.1774,  0.7220, -1.3295],
        [ 1.0285, -1.2779, -0.6747,  0.9241]])
y_custom:  tensor([[-1.6638,  0.1490,  0.5814,  0.9334],
        [ 0.3720, -0.6668, -1.1513,  1.4462],
        [-0.2171, -0.5644,  1.6809, -0.8994]])
```
Manual Layer Norm over 4D Tensor
```
import torch

batch_size, c, h, w = 2, 3, 2, 4
last_dims = [c, h, w]

embedding = torch.randn(batch_size, c, h, w)
print("x: ", embedding)

layer_norm = torch.nn.LayerNorm(last_dims, elementwise_affine = False)
layer_norm_out = layer_norm(embedding)
print("y: ", layer_norm_out)

eps: float = 0.00001
mean = torch.mean(embedding[0, :, :], dim=(-3, -2, -1), keepdim=True)
var = torch.square(embedding[0, :, :] - mean).mean(dim=(-3, -2, -1), keepdim=True)
y_custom = (embedding[0, :, :] - mean) / torch.sqrt(var + eps)
print("y_custom: ", y_custom)
assert torch.allclose(layer_norm_out[0], y_custom), 'Tensors do not match.'

eps: float = 0.00001
mean = torch.mean(embedding[1, :, :], dim=(-3, -2, -1), keepdim=True)
var = torch.square(embedding[1, :, :] - mean).mean(dim=(-3, -2, -1), keepdim=True)
y_custom = (embedding[1, :, :] - mean) / torch.sqrt(var + eps)
print("y_custom: ", y_custom)
assert torch.allclose(layer_norm_out[1], y_custom), 'Tensors do not match.'
```
Output
```
x:  tensor([[[[ 1.0902, -0.8648,  1.5785,  0.3087],
          [ 0.0249, -1.3477, -0.9565, -1.5024]],

         [[ 1.8024, -0.2894,  0.7284,  0.7822],
          [ 1.4385, -0.2848, -0.3114,  0.4633]],

         [[ 0.9061,  0.3066,  0.9916,  0.9284],
          [ 0.3356,  0.9162, -0.4579,  1.0669]]],

        [[[-0.8292,  0.9111, -0.7307, -1.1003],
          [ 0.3441, -1.9823,  0.1313,  0.2048]],

         [[-0.2838,  0.1147, -0.1605, -0.4637],
          [-2.1343, -0.4402,  1.6685,  0.4455]],

         [[ 0.6895, -2.7331,  1.1693, -0.6999],
          [-0.3497, -0.2942, -0.0028, -1.3541]]]])
y:  tensor([[[[ 0.8653, -1.3279,  1.4131, -0.0114],
          [-0.3298, -1.8697, -1.4309, -2.0433]],

         [[ 1.6643, -0.6824,  0.4594,  0.5198],
          [ 1.2560, -0.6772, -0.7071,  0.1619]],

         [[ 0.6587, -0.0137,  0.7547,  0.6838],
          [ 0.0188,  0.6701, -0.8715,  0.8392]]],

        [[[-0.4938,  1.2220, -0.3967, -0.7610],
          [ 0.6629, -1.6306,  0.4531,  0.5256]],

         [[ 0.0439,  0.4368,  0.1655, -0.1335],
          [-1.7805, -0.1103,  1.9686,  0.7629]],

         [[ 1.0035, -2.3707,  1.4764, -0.3663],
          [-0.0211,  0.0337,  0.3210, -1.0112]]]])
y_custom:  tensor([[[ 0.8653, -1.3279,  1.4131, -0.0114],
         [-0.3298, -1.8697, -1.4309, -2.0433]],

        [[ 1.6643, -0.6824,  0.4594,  0.5198],
         [ 1.2560, -0.6772, -0.7071,  0.1619]],

        [[ 0.6587, -0.0137,  0.7547,  0.6838],
         [ 0.0188,  0.6701, -0.8715,  0.8392]]])
y_custom:  tensor([[[-0.4938,  1.2220, -0.3967, -0.7610],
         [ 0.6629, -1.6306,  0.4531,  0.5256]],

        [[ 0.0439,  0.4368,  0.1655, -0.1335],
         [-1.7805, -0.1103,  1.9686,  0.7629]],

        [[ 1.0035, -2.3707,  1.4764, -0.3663],
         [-0.0211,  0.0337,  0.3210, -1.0112]]])
```
Example of custom layer norm implementation
```
from typing import Union, List

import torch

batch_size, seq_size, embed_dim = 2, 3, 4
embedding = torch.randn(batch_size, seq_size, embed_dim)
print("x: ", embedding)
print(embedding.shape)
print()

layer_norm = torch.nn.LayerNorm(embed_dim, elementwise_affine=False)
layer_norm_output = layer_norm(embedding)
print("y: ", layer_norm_output)
print(layer_norm_output.shape)
print()

def custom_layer_norm(
        x: torch.Tensor, dim: Union[int, List[int]] = -1, eps: float = 0.00001
) -> torch.Tensor:
    mean = torch.mean(x, dim=(dim,), keepdim=True)
    var = torch.square(x - mean).mean(dim=(dim,), keepdim=True)
    return (x - mean) / torch.sqrt(var + eps)

custom_layer_norm_output = custom_layer_norm(embedding)
print("y_custom: ", custom_layer_norm_output)
print(custom_layer_norm_output.shape)

assert torch.allclose(layer_norm_output, custom_layer_norm_output), 'Tensors do not match.'
```
Output
```
x:  tensor([[[-0.4808, -0.1981,  0.4538, -1.2653],
         [ 0.3578,  0.6592,  0.2161,  0.3852],
         [ 1.2184, -0.4238, -0.3415, -0.3487]],

        [[ 0.9874, -1.7737,  0.1886,  0.0448],
         [-0.5162,  0.7872, -0.3433, -0.3266],
         [-0.5459, -0.0371,  1.2625, -1.6030]]])
torch.Size([2, 3, 4])

y:  tensor([[[-0.1755,  0.2829,  1.3397, -1.4471],
         [-0.2916,  1.5871, -1.1747, -0.1208],
         [ 1.7301, -0.6528, -0.5334, -0.5439]],

        [[ 1.1142, -1.6189,  0.3235,  0.1812],
         [-0.8048,  1.7141, -0.4709, -0.4384],
         [-0.3057,  0.1880,  1.4489, -1.3312]]])
torch.Size([2, 3, 4])

y_custom:  tensor([[[-0.1755,  0.2829,  1.3397, -1.4471],
         [-0.2916,  1.5871, -1.1747, -0.1208],
         [ 1.7301, -0.6528, -0.5334, -0.5439]],

        [[ 1.1142, -1.6189,  0.3235,  0.1812],
         [-0.8048,  1.7141, -0.4709, -0.4384],
         [-0.3057,  0.1880,  1.4489, -1.3312]]])
torch.Size([2, 3, 4])
```


Question:
How to understand creating leaf tensors in PyTorch? From PyTorch documentation:
```
b = torch.rand(10, requires_grad=True).cuda()
b.is_leaf
False
# b was created by the operation that cast a cpu Tensor into a cuda Tensor

e = torch.rand(10).cuda().requires_grad_()
e.is_leaf
True
# e requires gradients and has no operations creating it

f = torch.rand(10, requires_grad=True, device="cuda")
f.is_leaf
True
# f requires grad, has no operation creating it
```
But why are e and f leaf Tensors, when they both were also cast from a CPU Tensor, into a Cuda Tensor (an operation)? Is it because Tensor e was cast into Cuda before the in-place operation requires_grad_()? And because f was cast by assignment device="cuda" rather than by method .cuda()?

Response:
When a tensor is first created, it becomes a leaf node. Basically, all inputs and weights of a neural network are leaf nodes of the computational graph. When any operation is performed on a tensor, it is not a leaf node anymore.
```
b = torch.rand(10, requires_grad=True) # create a leaf node
b.is_leaf # True
b = b.cuda() # perform a casting operation
b.is_leaf # False
```
requires_grad_() is not an operation in the same way as cuda() or others are. It creates a new tensor, because tensor which requires gradient (trainable weight) cannot depend on anything else.
```
e = torch.rand(10) # create a leaf node
e.is_leaf # True
e = e.cuda() # perform a casting operation
e.is_leaf # False
e = e.requires_grad_() # this creates a NEW tensor
e.is_leaf # True
```
Also, detach() operation creates a new tensor which does not require gradient:
```
b = torch.rand(10, requires_grad=True)
b.is_leaf # True
b = b.detach()
b.is_leaf # True
```
In the last example we create a new tensor which is already on a cuda device. We do not need any operation to cast it.
```
f = torch.rand(10, requires_grad=True, device="cuda") # create a leaf node on cuda
```


Question: PyTorch model to C++ I have trained the detection algorithm and saved my best model. Now I want to convert my model (pretrained) to C++ and use it in my app. I wanted to know what are the possible ways to convert a pyTorch model to c++? Thanks!

Response: You can use TorchScript intermediate representation of a PyTorch model, through tracing and scripting, that can be run in C++ environment. For this, you'll probably have to modify the model itself in order for it to be traced or scripted. You can use ONNX (Open Neural Network Exchange), through which you can export your model and load it in another C++ framework such as Caffe. It comes with its own implications though. The easiest is to try Embedding Python, through which you can run your python (pytorch) model in C++ environment. Note that the model will still run in python, but only through C++, so there won't be any speed gains that you might be expecting in C++. Also, with the release of torchvision 0.5, all models in torchvision have native support for TorchScript and ONNX.


Question: How to save model architecture in PyTorch? I know I can save a model by torch.save(model.state_dict(), FILE) or torch.save(model, FILE). But both of them don't save the architecture of model. So how can we save the architecture of a model in PyTorch like creating a .pb file in Tensorflow ? I want to apply different tweaks to my model. Do I have any better way than copying the whole class definition every time and creating a new class if I can't save the architecture of a model?

Response: Saving all the parameters (state_dict) and all the Modules is not enough, since there are operations that manipulates the tensors, but are only reflected in the actual code of the specific implementation (e.g., reshapeing in ResNet). Furthermore, the network might not have a fixed and pre-determined compute graph: You can think of a network that has branching or a loop (recurrence). Therefore, you must save the actual code. Alternatively, if there are no branches/loops in the net, you may save the computation graph, see, e.g., this post. You should also consider exporting your model using onnx and have a representation that captures both the trained weights as well as the computation graph.


Question:
PyTorch - How to use "toPILImage" correctly I would like to know, whether I used toPILImage from torchvision correctly. I want to use it, to see how the images look after initial image transformations are applied to the dataset. When I use it like in the code below, the image that comes up has weird colors like this one. The original image is a regular RGB image. This is my code:
```
import os
import torch
from PIL import Image, ImageFont, ImageDraw
import torch.utils.data as data
import torchvision
from torchvision import transforms
import matplotlib.pyplot as plt

# Image transformations
normalize = transforms.Normalize(
    mean=[0.485, 0.456, 0.406],
    std=[0.229, 0.224, 0.225]
    )
transform_img = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(256),
    transforms.ToTensor(),
    normalize ])

train_data = torchvision.datasets.ImageFolder(
    root='./train_cl/',
    transform=transform_img
    )
test_data = torchvision.datasets.ImageFolder(
    root='./test_named_cl/',
    transform=transform_img
    )

train_data_loader = data.DataLoader(train_data,
    batch_size=4,
    shuffle=True,
    num_workers=4) #num_workers=args.nThreads)

test_data_loader = data.DataLoader(test_data,
    batch_size=32,
    shuffle=False,
    num_workers=4)

# Open Image from dataset:
to_pil_image = transforms.ToPILImage()
my_img, _ = train_data[248]
results = to_pil_image(my_img)
results.show()
```
Edit: I had to use .data on the Torch Variable to get the tensor. Also I needed to rescale the numpy array before transposing. I found a working solution here, but it doesn't always work well. How can I do this better?
```
for i, data in enumerate(train_data_loader, 0):
    img, labels = data
    img = Variable(img)
    break

image = img.data.cpu().numpy()[0]

# This worked for rescaling:
image = (1/(2*2.25)) * image + 0.5

# Both of these didn't work:
# image /= (image.max()/255.0)
# image *= (255.0/image.max())

image = np.transpose(image, (1,2,0))
plt.imshow(image)
plt.show()
```

Response:
You can use PIL image but you're not actually loading the data as you would normally. Try something like this instead:
```
import numpy as np
import matplotlib.pyplot as plt

for img,labels in train_data_loader:
    # load a batch from train data
    break

# this converts it from GPU to CPU and selects first image
img = img.cpu().numpy()[0]
#convert image back to Height,Width,Channels
img = np.transpose(img, (1,2,0))
#show the image
plt.imshow(img)
plt.show()
```
As an update (02-10-2021):
```
import torchvision.transforms.functional as F
# load the image (creating a random image as an example)
img_data = torch.ByteTensor(4, 4, 3).random_(0, 255).numpy()
pil_image = F.to_pil_image(img_data)
```
Alternatively
```
import torchvision.transforms as transforms
img_data = torch.ByteTensor(4, 4, 3).random_(0, 255).numpy()
pil_image = transforms.ToPILImage()(img_data)
```
The second form can be integrated with dataset loader in pytorch or called directly as so. I added a modified to_pil_image here essentially it does what I suggested back in 2018 but it is integrated into pytorch now.


Question:
pytorch Network.parameters() missing 1 required positional argument: 'self' There's a problem when I call Network.parameters() in pytorch in this line in my main function: optimizer = optim.SGD(Network.parameters(), lr=0.001, momentum=0.9) I get the error code: TypeError: parameters() missing 1 required positional argument: 'self' My network is defined in this class
```
class Network(nn.Module):
def __init__(self):
    super(Network, self).__init__()
    self.conv1 = nn.Conv2d(1, 32, 5)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(32, 64, 5)
    self.pool2 = nn.MaxPool2d(2, 2)
    self.conv3 = nn.Conv2d(64, 64, 5)
    self.pool2 = nn.MaxPool2d(2, 2)
    self.fc1 = nn.Linear(64 * 5 * 5, 512)
    self.fc2 = nn.Linear(512, 640)
    self.fc3 = nn.Linear(640, 3756)

def forward(self, x):
    x = self.pool(F.relu(self.conv(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = self.pool(F.relu(self.conv3(x)))
    x = x.view(-1, 64 * 5 * 5)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x
```
Pretty sure that I imported all torch modules correctly. Any ideas of what I'm doing wrong here? Thank you!

Response:
When doing Network.parameters() you are calling the static method parameters. But, parameters is an instance method. So you have to instansiate Network before calling parameters.
```
network = Network()
optimizer = optim.SGD(network.parameters(), lr=0.001, momentum=0.9)
```
Or, if you only needs Network first this particular line:
```
optimizer = optim.SGD(Network().parameters(), lr=0.001, momentum=0.9)
```


Question:
Variational Autoencoder gives same output image for every input mnist image when using KL divergence When not using KL divergence term, the VAE reconstructs mnist images almost perfectly but fails to generate new ones properly when provided with random noise. When using KL divergence term, the VAE gives the same weird output both when reconstructing and generating images. Here's the pytorch code for the loss function:
```python
def loss_function(recon_x, x, mu, logvar):
    BCE = F.binary_cross_entropy(recon_x, x.view(-1, 784), size_average=True)
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return (BCE+KLD)
```
recon_x is the reconstructed image, x is the original_image, mu is the mean vector while logvar is the vector containing the log of variance. What is going wrong here? Thanks in advance :)

Response: A possible reason is the numerical unbalance between the two losses, with your BCE loss computed as an average over the batch (c.f. size_average=True) while the KLD one is summed.


Question:
`return_sequences = False` equivalent in pytorch LSTM In tensorflow/keras, we can simply set return_sequences = False for the last LSTM layer before the classification/fully connected/activation (softmax/sigmoid) layer to get rid of the temporal dimension. In PyTorch, I don't find anything similar. For the classification task, I don't need a sequence to sequence model but many to one architecture like this: Here's my simple bi-LSTM model.
```
import torch
from torch import nn

class BiLSTMClassifier(nn.Module):
    def __init__(self):
        super(BiLSTMClassifier, self).__init__()
        self.embedding = torch.nn.Embedding(num_embeddings = 65000, embedding_dim = 64)
        self.bilstm = torch.nn.LSTM(input_size = 64, hidden_size = 8, num_layers = 2,
                                    batch_first = True, dropout = 0.2, bidirectional = True)
        # as we have 5 classes
        self.linear = nn.Linear(8*2*512, 5) # last dimension
    def forward(self, x):
        x = self.embedding(x)
        print(x.shape)
        x, _ = self.bilstm(x)
        print(x.shape)
        x = self.linear(x.reshape(x.shape[0], -1))
        print(x.shape)

# create our model

bilstmclassifier = BiLSTMClassifier()
```
If I observe the shapes after each layer,
```
xx = torch.tensor(X_encoded[0]).reshape(1,512)
print(xx.shape)
# torch.Size([1, 512])
bilstmclassifier(xx)
#torch.Size([1, 512, 64])
#torch.Size([1, 512, 16])
#torch.Size([1, 5])
```
What can I do so that the last LSTM returns a tensor with shape (1, 16) instead of (1, 512, 16)?

Response:
The simplest way to do this is by indexing into the tensor:
```
x = x[:, -1, :]
```
where x is the RNN output. Of course, if batch_first is False, one would have to use x[-1, :, :] (or just x[-1]) to index into the time axis instead. Turns out this is the same thing Tensorflow/Keras do. The relevant code can be found in K.rnn here:
```
last_output = tuple(o[-1] for o in outputs)
```
Note that the code at this point uses time_major data format, so the index is into the first axis. Also, outputs is a tuple because it can be multiple layers, state/cell pairs etc., but it is generally the sequence of outputs for all time steps. This is then used in the RNN class as follows:
```
if self.return_sequences:
    output = K.maybe_convert_to_ragged(is_ragged_input, outputs, row_lengths)
else:
    output = last_output
```
So in total, we can see that return_sequences=False just uses outputs[-1].


Question:
Repeating a pytorch tensor without copying memory Does pytorch support repeating a tensor without allocating significantly more memory? Assume we have a tensor
```
t = torch.ones((1,1000,1000))
t10 = t.repeat(10,1,1)
```
Repeating t 10 times will require take 10x the memory. Is there a way how I can create a tensor t10 without allocating significantly more memory? Here is a related question, but without answers.

Response:
You can use torch.expand
```
t = torch.ones((1, 1000, 1000))
t10 = t.expand(10, 1000, 1000)
```
Keep in mind that the t10 is just a reference to t. So for example, a change to t10[0,0,0] will result in the same change in t[0,0,0] and every member of t10[:,0,0]. Other than direct access, most operations performed on t10 will cause memory to be copied which will break the reference and cause more memory to be used. For example: changing the device (.cpu(), .to(device=...), .cuda()), changing the datatype (.float(), .long(), .to(dtype=...)), or using .contiguous().


Question:
PyTorch: Learning rate scheduler How do I use a learning rate scheduler with the following optimizer?
```
optimizer = torch.optim.Adam(optim_params,betas=(args.momentum, args.beta), weight_decay=args.weight_decay)
```
I have written the following scheduler:
```
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.9)
```
I am not sure about whether I should step the scheduler or the optimizer. Which order should I take to perform the following?
```
optimizer.zero_grad()
scheduler.step()
optimizer.step()
```

Response:
Since 1.3 the behaviour was changed, see releases and this issue especially. Before this version, you should step scheduler before optimizer, which IMO wasn't reasonable. There was some back and forth (actually it breaks backward compatibility and IMO it's not a good idea to break it for such a minor inconvenience), but currently you should step scheduler after optimizer.
```
optimizer.zero_grad()
optimizer.step()
scheduler.step()
```


Question:
Delete an element from torch.Tensor I'm trying to delete an item from a tensor. In the example below, How can I remove the third item from the tensor ?
```
tensor([[-5.1949, -6.2621, -6.2051, -5.8983, -6.3586, -6.2434, -5.8923, -6.1901,
         -6.5713, -6.2396, -6.1227, -6.4196, -3.4311, -6.8903, -6.1248, -6.3813,
         -6.0152, -6.7449, -6.0523, -6.4341, -6.8579, -6.1961, -6.5564, -6.6520,
         -5.9976, -6.3637, -5.7560, -6.7946, -5.4101, -6.1310, -3.3249, -6.4584,
         -6.2202, -6.3663, -6.9293, -6.9262]], grad_fn=<SqueezeBackward1>)
```

Response:
I think that doing this with indexing is more readable.
```
t[t!=t[0,3]]
```
The result is the same as with the cat solution from below. BE CAREFUL: This will usually work for floats, but beware that if the value at [0,3] occurs more than once in the array, you will remove all occurrences of this item.


Question: How can I get the MSE of a tensor across a specific dimension? I have 2 tensors with .size of torch.Size([2272, 161]). I want to get mean-squared-error between them. However, I want it along each of the 161 channels, so that my error tensor has a .size of torch.Size([161]). How can I accomplish this? It seems that torch.nn.MSELoss doesn't let me specify a dimension.

Response:
For the nn.MSELoss you can specify the option reduction='none'. This then gives you back the squared error for each entry position of both of your tensors. Then you can apply torch.sum/torch.mean.
```
a = torch.randn(2272,161)
b = torch.randn(2272,161)
loss = nn.MSELoss(reduction='none')
loss_result = torch.sum(loss(a,b),dim=0)
```
I don't think there is a direct way to specify at the initialisation of the loss to which dimension to apply mean/sum. Hope that helps!


Question:
Converting state-parameters of Pytorch LSTM to Keras LSTM I was trying to port an existing trained PyTorch model into Keras. During the porting, I got stuck at LSTM layer. Keras implementation of LSTM network seems to have three state kind of state matrices while Pytorch implementation have four. For eg, for an Bidirectional LSTM with hidden_layers=64, input_size=512 & output size=128 state parameters where as follows State params of Keras LSTM
```
[<tf.Variable 'bidirectional_1/forward_lstm_1/kernel:0' shape=(512, 256) dtype=float32_ref>,
 <tf.Variable 'bidirectional_1/forward_lstm_1/recurrent_kernel:0' shape=(64, 256) dtype=float32_ref>,
 <tf.Variable 'bidirectional_1/forward_lstm_1/bias:0' shape=(256,) dtype=float32_ref>,
 <tf.Variable 'bidirectional_1/backward_lstm_1/kernel:0' shape=(512, 256) dtype=float32_ref>,
 <tf.Variable 'bidirectional_1/backward_lstm_1/recurrent_kernel:0' shape=(64, 256) dtype=float32_ref>,
 <tf.Variable 'bidirectional_1/backward_lstm_1/bias:0' shape=(256,) dtype=float32_ref>]
```
State params of PyTorch LSTM
```
['rnn.0.rnn.weight_ih_l0', torch.Size([256, 512])],
 ['rnn.0.rnn.weight_hh_l0', torch.Size([256, 64])],
 ['rnn.0.rnn.bias_ih_l0', torch.Size([256])],
 ['rnn.0.rnn.bias_hh_l0', torch.Size([256])],
 ['rnn.0.rnn.weight_ih_l0_reverse', torch.Size([256, 512])],
 ['rnn.0.rnn.weight_hh_l0_reverse', torch.Size([256, 64])],
 ['rnn.0.rnn.bias_ih_l0_reverse', torch.Size([256])],
 ['rnn.0.rnn.bias_hh_l0_reverse', torch.Size([256])],
```
I tried to look in to the code of both implementation but not able to understand much. Can someone please help me to transform 4-set of state params from PyTorch into 3-set of state params in Keras

Response:
They are really not that different. If you sum up the two bias vectors in PyTorch, the equations will be the same as what's implemented in Keras. This is the LSTM formula on PyTorch documentation: PyTorch uses two separate bias vectors for the input transformation (with a subscript starts with i) and recurrent transformation (with a subscript starts with h). In Keras LSTMCell:
```py
x_i = K.dot(inputs_i, self.kernel_i)
        x_f = K.dot(inputs_f, self.kernel_f)
        x_c = K.dot(inputs_c, self.kernel_c)
        x_o = K.dot(inputs_o, self.kernel_o)
        if self.use_bias:
            x_i = K.bias_add(x_i, self.bias_i)
            x_f = K.bias_add(x_f, self.bias_f)
            x_c = K.bias_add(x_c, self.bias_c)
            x_o = K.bias_add(x_o, self.bias_o)

        if 0 < self.recurrent_dropout < 1.:
            h_tm1_i = h_tm1 * rec_dp_mask[0]
            h_tm1_f = h_tm1 * rec_dp_mask[1]
            h_tm1_c = h_tm1 * rec_dp_mask[2]
            h_tm1_o = h_tm1 * rec_dp_mask[3]
        else:
            h_tm1_i = h_tm1
            h_tm1_f = h_tm1
            h_tm1_c = h_tm1
            h_tm1_o = h_tm1
        i = self.recurrent_activation(x_i + K.dot(h_tm1_i,
                                                  self.recurrent_kernel_i))
        f = self.recurrent_activation(x_f + K.dot(h_tm1_f,
                                                  self.recurrent_kernel_f))
        c = f * c_tm1 + i * self.activation(x_c + K.dot(h_tm1_c,
                                                        self.recurrent_kernel_c))
        o = self.recurrent_activation(x_o + K.dot(h_tm1_o,
                                                  self.recurrent_kernel_o))
```
There's only one bias added in the input transformation. However, the equations would be equivalent if we sum up the two biases in PyTorch. The two-bias LSTM is what's implemented in cuDNN (see the developer guide). I'm really not that familiar with PyTorch, but I guess that's why they use two bias parameters. In Keras, the CuDNNLSTM layer also has two bias weight vectors.


Question:
How to set gradients to Zero without optimizer? Between mutliple .backward() passes I'd like to set the gradients to zero. Right now I have to do this for every component seperately (here these are x and t), is there a way to do this "globally" for all affected variables? (I imagine something like z.set_all_gradients_to_zero().) I know there is optimizer.zero_grad() if you use an optimizer, but is there also a direct way without using an optimizer?
```
import torch

x = torch.randn(3, requires_grad = True)
t = torch.randn(3, requires_grad = True)
y = x + t
z = y + y.flip(0)

z.backward(torch.tensor([1., 0., 0.]), retain_graph = True)
print(x.grad)
print(t.grad)
x.grad.data.zero_()  # both gradients need to be set to zero
t.grad.data.zero_()
z.backward(torch.tensor([0., 1., 0.]), retain_graph = True)
print(x.grad)
print(t.grad)
```

Response:
You can also use nn.Module.zero_grad(). In fact, optim.zero_grad() just calls nn.Module.zero_grad() on all parameters which were passed to it. There is no reasonable way to do it globally. You can collect your variables in a list
```
grad_vars = [x, t]
for var in grad_vars:
    var.grad = None
```
or create some hacky function based on vars(). Perhaps it's also possible to inspect the computation graph and zero the gradient of all leaf nodes, but I am not familiar with the graph API. Long story short, you're expected to use the object-oriented interface of torch.nn instead of manually creating tensor variables.


Question:
Comparing Conv2D with padding between Tensorflow and PyTorch I am trying to import weights saved from a Tensorflow model to PyTorch. So far the results have been very similar. I ran into a snag when the model calls for conv2d with stride=2. To verify the mismatch, I set up a very simple comparison between TF and PyTorch. First, I compare conv2d with stride=1.
```
import tensorflow as tf
import numpy as np
import torch
import torch.nn.functional as F

np.random.seed(0)
sess = tf.Session()

# Create random weights and input
weights = torch.empty(3, 3, 3, 8)
torch.nn.init.constant_(weights, 5e-2)
x = np.random.randn(1, 3, 10, 10)

weights_tf = tf.convert_to_tensor(weights.numpy(), dtype=tf.float32)
# PyTorch adopts [outputC, inputC, kH, kW]
weights_torch = torch.Tensor(weights.permute((3, 2, 0, 1)))

# Tensorflow defaults to NHWC
x_tf = tf.convert_to_tensor(x.transpose((0, 2, 3, 1)), dtype=tf.float32)
x_torch = torch.Tensor(x)

# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, 1, 1, 1],
                         padding="SAME")

# PyTorch Conv2D
torch_conv2d = F.conv2d(x_torch, weights_torch, padding=1, stride=1)

sess.run(tf.global_variables_initializer())
tf_result = sess.run(tf_conv2d)

diff = np.mean(np.abs(tf_result.transpose((0, 3, 1, 2)) - torch_conv2d.detach().numpy()))
print('Mean of Abs Diff: {0}'.format(diff))
```
The result of this execution is:
```
Mean of Abs Diff: 2.0443112092038973e-08
```
When I change stride to 2, the results start to vary.
```
# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, 2, 2, 1],
                         padding="SAME")

# PyTorch Conv2D
torch_conv2d = F.conv2d(x_torch, weights_torch, padding=1, stride=2)
```
The result of this execution is:
```
Mean of Abs Diff: 0.2104552686214447
```
According to PyTorch documentation, conv2d uses zero-padding defined by the padding argument. Thus, zeros are added to the left, top, right, and bottom of the input in my example. If PyTorch simply adds padding on both sides based on the input parameter, it should be easy to replicate in Tensorflow.
```
# Manually add padding - consistent with PyTorch
paddings = tf.constant([[0, 0], [1, 1], [1, 1], [0, 0]])
x_tf = tf.convert_to_tensor(x.transpose((0, 2, 3, 1)), dtype=tf.float32)
x_tf = tf.pad(x_tf, paddings, "CONSTANT")

# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, 2, 2, 1],
                         padding="VALID")
```
The result of this comparison is:
```
Mean of Abs Diff: 1.6035047067930464e-08
```
What this tells me is that if I am somehow able to replicate the default padding behavior from Tensorflow into PyTorch, then my results will be similar. This question inspected the behavior of padding in Tensorflow. TF documentation explains how padding is added for "SAME" convolutions. I discovered these links while writing this question. Now that I know the padding strategy of Tensorflow, I can implement it in PyTorch.

Response:
To replicate the behavior, padding sizes are calculated as described in the Tensorflow documentation. Here, I test the padding behavior by setting stride=2 and padding the PyTorch input.
```
import tensorflow as tf
import numpy as np
import torch
import torch.nn.functional as F

np.random.seed(0)
sess = tf.Session()

# Create random weights and input
weights = torch.empty(3, 3, 3, 8)
torch.nn.init.constant_(weights, 5e-2)
x = np.random.randn(1, 3, 10, 10)

weights_tf = tf.convert_to_tensor(weights.numpy(), dtype=tf.float32)
weights_torch = torch.Tensor(weights.permute((3, 2, 0, 1)))

# Tensorflow padding behavior. Assuming that kH == kW to keep this simple.
stride = 2
if x.shape[2] % stride == 0:
    pad = max(weights.shape[0] - stride, 0)
else:
    pad = max(weights.shape[0] - (x.shape[2] % stride), 0)

if pad % 2 == 0:
    pad_val = pad // 2
    padding = (pad_val, pad_val, pad_val, pad_val)
else:
    pad_val_start = pad // 2
    pad_val_end = pad - pad_val_start
    padding = (pad_val_start, pad_val_end, pad_val_start, pad_val_end)

x_tf = tf.convert_to_tensor(x.transpose((0, 2, 3, 1)), dtype=tf.float32)
x_torch = torch.Tensor(x)
x_torch = F.pad(x_torch, padding, "constant", 0)

# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, stride, stride, 1],
                         padding="SAME")

# PyTorch Conv2D
torch_conv2d = F.conv2d(x_torch, weights_torch, padding=0, stride=stride)

sess.run(tf.global_variables_initializer())
tf_result = sess.run(tf_conv2d)

diff = np.mean(np.abs(tf_result.transpose((0, 3, 1, 2)) - torch_conv2d.detach().numpy()))
print('Mean of Abs Diff: {0}'.format(diff))
```
The output is:
```
Mean of Abs Diff: 2.2477470551507395e-08
```
I wasn't quite sure why this was happening when I started writing this question, but a bit of reading clarified this very quickly. I hope this example can help others.


Question: What is a buffer in Pytorch? I understand what register_buffer does and the difference between register_buffer and register_parameters. But what is the precise definition of a buffer in PyTorch?

Response:
This can be answered looking at the implementation:
```py
def register_buffer(self, name, tensor):
    if '_buffers' not in self.__dict__:
        raise AttributeError(
            "cannot assign buffer before Module.__init__() call")
    elif not isinstance(name, torch._six.string_classes):
        raise TypeError("buffer name should be a string. "
                        "Got {}".format(torch.typename(name)))
    elif '.' in name:
        raise KeyError("buffer name can't contain \".\"")
    elif name == '':
        raise KeyError("buffer name can't be empty string \"\"")
    elif hasattr(self, name) and name not in self._buffers:
        raise KeyError("attribute '{}' already exists".format(name))
    elif tensor is not None and not isinstance(tensor, torch.Tensor):
        raise TypeError("cannot assign '{}' object to buffer '{}' "
                        "(torch Tensor or None required)"
                        .format(torch.typename(tensor), name))
    else:
        self._buffers[name] = tensor
```
That is, the buffer's name: must be a string: not isinstance(name, torch._six.string_classes) cannot contain a . (dot): '.' in name cannot be an empty string: name == '' cannot be an attribute of the Module: hasattr(self, name) should be unique: name not in self._buffers and the tensor (guess what?): should be a Tensor: isinstance(tensor, torch.Tensor) So, the buffer is just a tensor with these properties, registered in the _buffers attribute of a Module;


Question: What is the backward process of max operation in deep learning? I know that the backward process of deep learning follows the gradient descent algorithm. However, there is never a gradient concept for max operation. How does deep learning frameworks like tensorflow, pytorch deal with the backward of 'max' operation like maxpooling?

Response:
You have to think of what the max operator actually does? That is: It returns or lets better say it propagates the maximum. And that's exactly what it does here - it takes two or more tensors and propagates forward (only) the maximum. It is often helpful to take a look at a short example:
```py
t1 = torch.rand(10, requires_grad=True)
t2 = torch.rand(10, requires_grad=True)

s1 = torch.sum(t1)
s2 = torch.sum(t2)
print('sum t1:', s1, 'sum t2:', s2)
m = torch.max(s1, s2)
print('max:', m, 'requires_grad:', m.requires_grad)
m.backward()
print('t1 gradients:', t1.grad)
print('t2 gradients:', t2.grad)
```
This code creates two random tensors sums them up and puts them through a max function. Then backward() is called upon the result. Lets take a look at the two possible outcomes: Outcome 1 - sum of t1 is larger:
```py
sum t1: tensor(5.6345) sum t2: tensor(4.3965)
max: tensor(5.6345) requires_grad: True
t1 gradients: tensor([ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.])
t2 gradients: tensor([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])
```
Outcome 2 - sum of t2 is larger:
```py
sum t1: tensor(3.3263) sum t2: tensor(4.0517)
max: tensor(4.0517) requires_grad: True
t1 gradients: tensor([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])
t2 gradients: tensor([ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.])
```
As you would expect in the case s1 represents the maximum gradients will be calculated for t1. Similarly when s2 is the maximum gradients will be calculated for t2. Similarly like in the forward step, back propagation is propagating backwards through the maximum. One thing worth mentioning is that the other tensors which do not represent the maximum are still part of the graph. Only the gradients are set to zero then. If they wouldn't be part of the graph you would get None as gradient, instead of a zero vector. You can check what happens if you use python-max instead of torch.max:
```py
t1 = torch.rand(10, requires_grad=True)
t2 = torch.rand(10, requires_grad=True)

s1 = torch.sum(t1)
s2 = torch.sum(t2)
print('sum t1:', s1, 'sum t2:', s2)
m = max(s1, s2)
print('max:', m, 'requires_grad:', m.requires_grad)
m.backward()
print('t1 gradients:', t1.grad)
print('t2 gradients:', t2.grad)
```
Output:
```py
sum t1: tensor(4.7661) sum t2: tensor(4.4166)
max: tensor(4.7661) requires_grad: True
t1 gradients: tensor([ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.])
t2 gradients: None
```


Question:
How to check the output gradient by each layer in pytorch in my code? I am working on the pytorch to learn. And There is a question how to check the output gradient by each layer in my code. My code is below
```
#import the nescessary libs
import numpy as np
import torch
import time

# Loading the Fashion-MNIST dataset
from torchvision import datasets, transforms

# Get GPU Device

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Define a transform to normalize the data
transform = transforms.Compose([transforms.ToTensor(),
                                    transforms.Normalize((0.5,), (0.5,))
                                                                   ])
# Download and load the training data
trainset = datasets.FashionMNIST('MNIST_data/', download = True, train = True, transform = transform)
testset = datasets.FashionMNIST('MNIST_data/', download = True, train = False, transform = transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size = 32, shuffle = True, num_workers=4)
testloader = torch.utils.data.DataLoader(testset, batch_size = 32, shuffle = True, num_workers=4)

# Examine a sample
dataiter = iter(trainloader)
images, labels = dataiter.next()

# Define the network architecture
from torch import nn, optim
import torch.nn.functional as F

model = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 10),
                      nn.LogSoftmax(dim = 1)
                     )
model.to(device)

# Define the loss
criterion = nn.CrossEntropyLoss()

# Define the optimizer
optimizer = optim.Adam(model.parameters(), lr = 0.001)

# Define the epochs
epochs = 5

train_losses, test_losses = [], []

# start = time.time()
for e in range(epochs):
    running_loss = 0

    for images, labels in trainloader:
    # Flatten Fashion-MNIST images into a 784 long vector
        images = images.to(device)
        labels = labels.to(device)
        images = images.view(images.shape[0], -1)

    # Training pass
        optimizer.zero_grad()

        output = model.forward(images)
        loss = criterion(output, labels)

        loss.backward()

#         print(loss.grad)

        optimizer.step()

        running_loss += loss.item()

    else:
        print(model[0].grad)
```
If I print model[0].grad after back-propagation, Is it going to be the output gradient by each layer for every epoches? Or, If I want to know the output gradient by each layer, where and what am I should print? Thank you!! Thank you for reading

Response:
Well, this is a good question if you need to know the inner computation within your model. Let me explain to you! So firstly when you print the model variable you'll get this output:
```
Sequential(
  (0): Linear(in_features=784, out_features=128, bias=True)
  (1): ReLU()
  (2): Linear(in_features=128, out_features=10, bias=True)
  (3): LogSoftmax(dim=1)
)
```
And if you choose model[0], that means you have selected the first layer of the model. that is Linear(in_features=784, out_features=128, bias=True). If you will look at the documentation of torch.nn.Linear here, you will find that there are two variables to this class that you can access. One is Linear.weight and the other is Linear.bias which will give you the weights and biases of that corresponding layer respectively. Remember you cannot use model.weight to look at the weights of the model as your linear layers are kept inside a container called nn.Sequential which doesn't has a weight attribute. So coming back to looking at weights and biases, you can access them per layer. So model[0].weight and model[0].bias are the weights and biases of the first layer. And similarly to access the gradients of the first layer model[0].weight.grad and model[0].bias.grad will be the gradients.


Question: Multiply all elements of PyTorch tensor I wanted to do something like this question in PyTorch i.e. multiply all elements with each other keeping a certain axis constant. Does PyTorch has any pre-defined function for this?

Response:
Yes. torch.prod. Use the dim parameter to tell which along which axis you want the product to be computed.
```py
x = torch.randn((2, 2))
print(x)
print(torch.prod(x, 0)) # product along 0th axis
```
This prints
```
tensor([[-0.3661, 1.0693],
           [0.5144, 1.3489]])
tensor([-0.1883, 1.4424])
```


Question:
Performing Convolution (NOT cross-correlation) in pytorch I have a network that I am trying to implement in pytorch, and I cannot seem to figure out how to implement "pure" convolution. In tensorflow it could be accomplished like this:
```
def conv2d_flipkernel(x, k, name=None):
    return tf.nn.conv2d(x, flipkernel(k), name=name,
                        strides=(1, 1, 1, 1), padding='SAME')
```
With the flipkernel function being:
```
def flipkernel(kern):
      return kern[(slice(None, None, -1),) * 2 + (slice(None), slice(None))]
```
How can something similar be done in pytorch?

Response:
TLDR Use the convolution from the functional toolbox, torch.nn.fuctional.conv2d, not torch.nn.Conv2d, and flip your filter around the vertical and horizontal axis. torch.nn.Conv2d is a convolutional layer for a network. Because weights are learned, it does not matter if it is implemented using cross-correlation, because the network will simply learn a mirrored version of the kernel (Thanks @etarion for this clarification). torch.nn.fuctional.conv2d performs convolution with the inputs and weights provided as arguments, similar to the tensorflow function in your example. I wrote a simple test to determine whether, like the tensorflow function, it is actually performing cross-correlation and it is necessary to flip the filter for correct convolutional results.
```py
import torch
import torch.nn.functional as F
import torch.autograd as autograd
import numpy as np

#A vertical edge detection filter.
#Because this filter is not symmetric, for correct convolution the filter must be flipped before element-wise multiplication
filters = autograd.Variable(torch.FloatTensor([[[[-1, 1]]]]))

#A test image of a square
inputs = autograd.Variable(torch.FloatTensor([[[[0,0,0,0,0,0,0], [0, 0, 1, 1, 1, 0, 0],
                                             [0, 0, 1, 1, 1, 0, 0], [0, 0, 1, 1, 1, 0, 0],
                                            [0,0,0,0,0,0,0]]]]))
print(F.conv2d(inputs, filters))
```
This outputs
```none
Variable containing:
(0 ,0 ,.,.) =
  0  0  0  0  0  0
  0  1  0  0 -1  0
  0  1  0  0 -1  0
  0  1  0  0 -1  0
  0  0  0  0  0  0
[torch.FloatTensor of size 1x1x5x6]
```
This output is the result for cross-correlation. Therefore, we need to flip the filter
```py
def flip_tensor(t):
    flipped = t.numpy().copy()

    for i in range(len(filters.size())):
        flipped = np.flip(flipped,i) #Reverse given tensor on dimention i
    return torch.from_numpy(flipped.copy())

print(F.conv2d(inputs, autograd.Variable(flip_tensor(filters.data))))
```
The new output is the correct result for convolution.
```none
Variable containing:
(0 ,0 ,.,.) =
  0  0  0  0  0  0
  0 -1  0  0  1  0
  0 -1  0  0  1  0
  0 -1  0  0  1  0
  0  0  0  0  0  0
[torch.FloatTensor of size 1x1x5x6]
```


Question:
RuntimeError: 0D or 1D target tensor expected, multi-target not supported I was training a deep learning model but I am getting this issue
```py
*My Training Model*
def train(model,criterion,optimizer,iters):
    epoch = iters
    train_loss = []
    validaion_loss = []
    train_acc = []
    validation_acc = []
    states = ['Train','Valid']
    for epoch in range(epochs):
        print("epoch : {}/{}".format(epoch+1,epochs))
        for phase in states:
            if phase == 'Train':
                model.train() *training the data if phase is train*
                dataload = train_data_loader
            else:
                model.eval()
                dataload = valid_data_loader

            run_loss,run_acc = 0,0 *creating variables to calculate loss and acc*
            for data in dataload:
                inputs,labels = data
                inputs = inputs.to(device)
                labels = labels.to(device)

                labels = labels.byte()
                optimizer.zero_grad() #Using the optimizer

                with torch.set_grad_enabled(phase == 'Train'):
                    outputs = model(inputs)
                    loss = criterion(outputs,labels.unsqueeze(1).float())

                    predict = outputs>=0.5
                    if phase == 'Train':
                        loss.backward() #backward propagation
                        optimizer.step()

                    acc = torch.sum(predict == labels.unsqueeze(1))
                run_loss+=loss.item()
                run_acc+=acc.item()/len(labels)
            if phase == 'Train': #calulating train loss and accucracy
                epoch_loss = run_loss/len(train_data_loader)
                train_loss.append(epoch_loss)
                epoch_acc = run_acc/len(train_data_loader)
                train_acc.append(epoch_acc)
            else: #training validation loss and accuracy
                epoch_loss = run_loss/len(valid_data_loader)
                validaion_loss.append(epoch_loss)
                epoch_acc = run_acc/len(valid_data_loader)
                validation_acc.append(epoch_acc)

            print("{}, loss :{},accuracy:{}".format(phase,epoch_loss,epoch_acc))

    history = {'Train_loss':train_loss,'Train_accuracy':train_acc,
               'Validation_loss':validaion_loss,'Validation_Accuracy':validation_acc}
    return model,history[enter image description here][1]
```
I was experiencing the error as 0D or 1D target tensor expected, multi-target not supported could you please help in rectifying the code which is described above. Referred the previous related articles but unable to get the desired result. What are the code snippets I had to change so that my model will run successfully. Any suggestions are mostly welcome. Thanks in Advance.

Response: Your problem is that labels have the correct shape to calculate the loss. When you add .unsqueeze(1) to labels you made your labels with this shape [32,1] which is not consistent to the requirment to calcualte the loss. To fix the problem, you only need to remove .unsqueeze(1) for labels. If you read the documentation of CrossEntropLoss, the arguments: Input should be in (N,C) shape which is outputs in your case and [32,3]. Target should be in N shape which is labels in your case and should be [32]. Therefore, the loss function expects labels to be in 1D target not multi-target.


Question:
To calculate euclidean distance between vectors in a torch tensor with multiple dimensions There is a random initialized torch tensor of the shape as below. Inputs
```
tensor1 = torch.rand((4,2,3,100))
tensor2 = torch.rand((4,2,3,100))
```
tensor1 and tensor2 are torch tensors with 24 100-dimensional vectors, respectively. I want to get a tensor with a shape of torch.size([4,2,3]) by obtaining the Euclidean distance between vectors with the same index of two tensors. I used dist = torch.nn.functional.pairwise_distance(tensor1, tensor2) to get the results I wanted. However, the pairwise_distance function calculates the euclidean distance for the second dimension of the tensor. So dist shape is torch.size([4,3,100]). I have performed transpose several times to solve these problems. My code is as follows.
```
tensor1 = tensor1.transpose(1,3)
tensor2 = tensor2.transpose(1,3)
dist = torch.nn.functional.pairwise_distance(tensor1, tensor2)
dist = dist.transpose(1,2)
```
Is there a simpler or easier way to get the result I want?

Response:
Here ya go
```
dist = (tensor1 - tensor2).pow(2).sum(3).sqrt()
```
Basically that's what Euclidean distance is. Subtract -> power by 2 -> sum along the unfortunate axis you want to eliminate-> square root


Question:
Pytorch transform.ToTensor() changes image I want to convert images to tensor using torchvision.transforms.ToTensor(). After processing, I printed the image but the image was not right. Here is my code:
```
trans = transforms.Compose([
    transforms.ToTensor()])

demo = Image.open(img)
demo_img = trans(demo)
demo_array = demo_img.numpy()*255
print(Image.fromarray(demo_array.astype(np.uint8)))
```
The original image is: After processing, it looks like:

Response:
It seems that the problem is with the channel axis. If you look at torchvision.transforms docs, especially on ToTensor() Converts a PIL Image or numpy.ndarray (H x W x C) in the range [0, 255] to a torch.FloatTensor of shape (C x H x W) in the range [0.0, 1.0] So once you perform the transformation and return to numpy.array your shape is: (C, H, W) and you should change the positions, you can do the following:
```
demo_array = np.moveaxis(demo_img.numpy()*255, 0, -1)
```
This will transform the array to shape (H, W, C) and then when you return to PIL and show it will be the same image. So in total:
```
import numpy as np
from PIL import Image
from torchvision import transforms

trans = transforms.Compose([transforms.ToTensor()])

demo = Image.open(img)
demo_img = trans(demo)
demo_array = np.moveaxis(demo_img.numpy()*255, 0, -1)
print(Image.fromarray(demo_array.astype(np.uint8)))
```


Question:
Modifying a pytorch tensor and then getting the gradient lets the gradient not work I am a beginner in pytorch and I face the following issue: When I get the gradient of the below tensor (note that I use some variable x in some way as you can see below), I get the gradient:
```
import torch
myTensor = torch.randn(2, 2,requires_grad=True)
with torch.enable_grad():
    x=myTensor.sum() *10
x.backward()
print(myTensor.grad)
```
Now, if I try to modify an element of myTensor, I get the error of leaf variable has been moved into the graph interior. See this code:
```
import torch
myTensor = torch.randn(2, 2,requires_grad=True)
myTensor[0,0]*=5
with torch.enable_grad():
    x=myTensor.sum() *10
x.backward()
print(myTensor.grad)
```
What is wrong with my latter code? And how do I correct it? Any help would be highly appreciated. Thanks a lot!

Response:
The problem here is that this line represents an in-place operation:
```
myTensor[0,0]*=5
```
And PyTorch or more precisely autograd is not very good in handling in-place operations, especially on those tensors with the requires_grad flag set to True. You can also take a look here: https://pytorch.org/docs/stable/notes/autograd.html#in-place-operations-with-autograd Generally you should avoid in-place operations where it is possible, in some cases it can work, but you should always avoid in-place operations on tensors where you set requires_grad to True. Unfortunately there are not many pytorch functions to help out on this problem. So you would have to use a helper tensor to avoid the in-place operation in this case: Code:
```py
import torch

myTensor = torch.randn(2, 2,requires_grad=True)
helper_tensor = torch.ones(2, 2)
helper_tensor[0, 0] = 5
new_myTensor = myTensor * helper_tensor # new tensor, out-of-place operation
with torch.enable_grad():
    x=new_myTensor.sum() *10 # of course you need to use the new tensor
x.backward()                 # for further calculation and backward
print(myTensor.grad)
```
Output:
```
tensor([[50., 10.],
        [10., 10.]])
```
Unfortunately this is not very nice and I would appreciate if there would be a better or nicer solution out there. But for all I know in the current version (0.4.1) you will have to got with this workaround for tensors with gradient resp. requires_grad=True. Hopefully for future versions there will be a better solution. Btw. if you activate the gradient later you can see that it works just fine:
```
import torch
myTensor = torch.randn(2, 2,requires_grad=False) # no gradient so far
myTensor[0,0]*=5                                 # in-place op not included in gradient
myTensor.requires_grad = True                    # activate gradient here
with torch.enable_grad():
    x=myTensor.sum() *10
x.backward()                                     # no problem here
print(myTensor.grad)
```
But of course this will yield to a different result:
```
tensor([[10., 10.],
        [10., 10.]])
```
Hope this helps!


Question:
How to use random_split with percentage split (sum of input lengths does not equal the length of the input dataset) I tried to use torch.utils.data.random_split as follows:
```py
import torch
from torch.utils.data import DataLoader, random_split

list_dataset = [1,2,3,4,5,6,7,8,9,10]
dataset = DataLoader(list_dataset, batch_size=1, shuffle=False)

random_split(dataset, [0.8, 0.1, 0.1], generator=torch.Generator().manual_seed(123))
```
However, when I tried this, I got the error raise ValueError("Sum of input lengths does not equal the length of the input dataset!") I looked at the docs and it seems like I should be able to pass in decimals that sum to 1, but clearly it's not working. I also Googled this error and the closest thing that comes up is this issue. What am I doing wrong?

Response:
You're likely using an older version of PyTorch, such as Pytorch 1.10, which does not have this functionality. To replicate this functionality in the older version, you can just copy the source code of the newer version:
```py
import math
from torch import default_generator, randperm
from torch._utils import _accumulate
from torch.utils.data.dataset import Subset

def random_split(dataset, lengths,
                 generator=default_generator):
    r"""
    Randomly split a dataset into non-overlapping new datasets of given lengths.

    If a list of fractions that sum up to 1 is given,
    the lengths will be computed automatically as
    floor(frac * len(dataset)) for each fraction provided.

    After computing the lengths, if there are any remainders, 1 count will be
    distributed in round-robin fashion to the lengths
    until there are no remainders left.

    Optionally fix the generator for reproducible results, e.g.:

    >>> random_split(range(10), [3, 7], generator=torch.Generator().manual_seed(42))
    >>> random_split(range(30), [0.3, 0.3, 0.4], generator=torch.Generator(
    ...   ).manual_seed(42))

    Args:
        dataset (Dataset): Dataset to be split
        lengths (sequence): lengths or fractions of splits to be produced
        generator (Generator): Generator used for the random permutation.
    """
    if math.isclose(sum(lengths), 1) and sum(lengths) <= 1:
        subset_lengths: List[int] = []
        for i, frac in enumerate(lengths):
            if frac < 0 or frac > 1:
                raise ValueError(f"Fraction at index {i} is not between 0 and 1")
            n_items_in_split = int(
                math.floor(len(dataset) * frac)  # type: ignore[arg-type]
            )
            subset_lengths.append(n_items_in_split)
        remainder = len(dataset) - sum(subset_lengths)  # type: ignore[arg-type]
        # add 1 to all the lengths in round-robin fashion until the remainder is 0
        for i in range(remainder):
            idx_to_add_at = i % len(subset_lengths)
            subset_lengths[idx_to_add_at] += 1
        lengths = subset_lengths
        for i, length in enumerate(lengths):
            if length == 0:
                warnings.warn(f"Length of split at index {i} is 0. "
                              f"This might result in an empty dataset.")

    # Cannot verify that dataset is Sized
    if sum(lengths) != len(dataset):    # type: ignore[arg-type]
        raise ValueError("Sum of input lengths does not equal the length of the input dataset!")

    indices = randperm(sum(lengths), generator=generator).tolist()  # type: ignore[call-overload]
    return [Subset(dataset, indices[offset - length : offset]) for offset, length in zip(_accumulate(lengths), lengths)]
```


Question:
groupby aggregate mean in pytorch I have a 2D tensor:
```py
samples = torch.Tensor([
    [0.1, 0.1],    #-> group / class 1
    [0.2, 0.2],    #-> group / class 2
    [0.4, 0.4],    #-> group / class 2
    [0.0, 0.0]     #-> group / class 0
])
```
and a label for each sample corresponding to a class:
```py
labels = torch.LongTensor([1, 2, 2, 0])
```
so len(samples) == len(labels). Now I want to calculate the mean for each class / label. Because there are 3 classes (0, 1 and 2) the final vector should have dimension [n_classes, samples.shape[1]] So the expected solution should be:
```py
result == torch.Tensor([
    [0.1, 0.1],
    [0.3, 0.3], # -> mean of [0.2, 0.2] and [0.4, 0.4]
    [0.0, 0.0]
])
```
Question: How can this be done in pure pytorch (i.e. no numpy so that I can autograd) and ideally without for loops?

Response:
All you need to do is form an mxn matrix (m=num classes, n=num samples) which will select the appropriate weights, and scale the mean appropriately. Then you can perform a matrix multiplication between your newly formed matrix and the samples matrix. Given your labels, your matrix should be (each row is a class number, each class a sample number and its weight):
```
[[0.0000, 0.0000, 0.0000, 1.0000],
 [1.0000, 0.0000, 0.0000, 0.0000],
 [0.0000, 0.5000, 0.5000, 0.0000]]
```
Which you can form as follows:
```
M = torch.zeros(labels.max()+1, len(samples))
M[labels, torch.arange(len(samples)] = 1
M = torch.nn.functional.normalize(M, p=1, dim=1)
torch.mm(M, samples)
```
Output:
```
tensor([[0.0000, 0.0000],
        [0.1000, 0.1000],
        [0.3000, 0.3000]])
```
Note that the output means are correctly sorted in class order. Why does M[labels, torch.arange(len(samples))] = 1 work? This is performing a broadcast operation between the labels and the number of samples. Essentially, we are generating a 2D index for every element in labels: the first specifies which of the m classes it belongs to, and the second simply specifies its index position (from 1 to N). Another way would be top explicitly generate all the 2D indices:
```
twoD_indices = []
for count, label in enumerate(labels):
  twoD_indices.append((label, count))
```


Question: Batchnorm2d Pytorch - Why pass number of channels to batchnorm? Why do I need to pass the previous nummber of channels to the batchnorm? The batchnorm should normalize over each datapoint in the batch, why does it need to have the number of channels then ?

Response:
Batch normalisation has learnable parameters, because it includes an affine transformation. From the documentation of nn.BatchNorm2d: The mean and standard-deviation are calculated per-dimension over the mini-batches and γ and β are learnable parameter vectors of size C (where C is the input size). By default, the elements of γ are set to 1 and the elements of β are set to 0. Since the norm is calculated per channel, the parameters γ and β are vectors of size num_channels (one element per channel), which results in an individual scale and shift per channel. As with any other learnable parameter in PyTorch, they need to be created with a fixed size, hence you need to specify the number of channels
```py
batch_norm = nn.BatchNorm2d(10)

# γ
batch_norm.weight.size()
# => torch.Size([10])

# β
batch_norm.bias.size()
# => torch.Size([10])
```
Note: Setting affine=False does not use any parameters and the number of channels wouldn't be needed, but they are still required, in order to have a consistent interface.


Question: tf.cast equivalent in pytorch? I am new to PyTorch. TensorFlow has an API tf.cast() and tf.shape(). the tf.cast has specific purpose in TensorFlow, is there anything equivalent in torch? i have tensor x= tensor(shape(128,64,32,32)): tf.shape(x) create tensor of dimension 1 x.shape create the true dimension. i need to use tf.shape(x) in torch. tf.cast has a different role than just changing tensor dtype in torch. did anyone have equivalent API in torch/PyTorch.

Response:
Check out the PyTorch Documentation As they mentioned:
```py
print(x.dtype) # Prints "torch.int64", currently 64-bit integer type
x = x.type(torch.FloatTensor)
print(x.dtype) # Prints "torch.float32", now 32-bit float
print(x.float()) # Still "torch.float32"
print(x.type(torch.DoubleTensor)) # Prints "tensor([0., 1., 2., 3.], dtype=torch.float64)"
print(x.type(torch.LongTensor)) # Cast back to int-64, prints "tensor([0, 1, 2, 3])"
```


Question:
Can you accelerate torch DL training on anything other than "cuda" like "hip" or "OpenCL"? I've noticed that torch.device can accept a range of arguments, precisely cpu, cuda, mkldnn, opengl, opencl, ideep, hip, msnpu. However, when training deep learning models, I've only ever seen cuda or cpu being used. Very often the code looks something like this
```
if torch.cuda.is_available():
    device = torch.device("cuda")
else:
    device = torch.device("cpu")
```
I've never seen any of the others being used, and was wondering if they can be used and how. The latest MacBooks with an AMD graphic card I believe should be able to use "hip", but is that true? And will the training speed be similar to that of using one CUDA GPU? If not, what is the point in torch.device accepting so many options if they cannot actually be used?

Response: If you want to use a GPU for deep learning there is selection between CUDA and CUDA... More broad answer, yes there is AMD's hip and some OpenCL implementation: The is hip by AMD - CUDA like interface with ports of PyTorch, hipCaffe, TensorFlow, but AMD's hip/rocm is supported only on Linux - no Windows or Mac OS support by rocm provided Even if you want to use Linux with AMD GPU + ROCM, you have to stick to GCN desrete devices (i.e. cards like rx 580, Vega 56/64 or Radeon VII), there is no hip/rocm support for RDNA devices (a year since a release) and it does not look to be any time soon, APUs aren't supported as well by hip. Only one popular frameworks that supports OpenCL are Caffe and Keras+PlaidML. But Caffe's issues: Caffe seems have not being actively developed any more and somewhat outdated by today's standard Performance of Caffe OpenCL implementation is about 1/2 of what is provided by nVidia's cuDNN and AMD's MIOpen, but it works quite OK and I used it in many cases. Latest version had even grater performance hit https://github.com/BVLC/caffe/issues/6585 but at least you can run a version that works several changes behind Also Caffe/OpenCL works there are still some bugs I fixed manually for OpenCL over AMD. https://github.com/BVLC/caffe/issues/6239 Keras/Plaid-ML Keras on its own is much weaker framework in terms of ability to access lower level functionality PlaidML performance is still 1/2 - to 1/3 of optimized NVidia's cuDNN & AMD's MIOpen-ROCM - and slower that caffe OpenCL in the tests I did The future of non-TF backends for keras is not clear since 2.4 it requires TF... Bottom line: If you have GCN discrete AMD GPU and you run Linux you can use ROCM+Hip. Yet it isn't as stable as CUDA You can try OpenCL Caffe or Keras-PlaidML - it maybe slower and not as optimal as other solutions but have higher chances of making it work. Edit 2021-09-14: there is a new project dlprimitives: https://github.com/artyom-beilis/dlprimitives that has better performance than both Caffe-OpenCL and Keras - it is ~75% performance for training in comparison to Keras/TF2, however it is under early development and has at this point much more limited set of layers that Caffe/Keras-PlaidML The connection to pytorch is work in progress with some initial results: https://github.com/artyom-beilis/pytorch_dlprim Disclaimer: I'm the author of this project


Question:
How to get only specific classes from PyTorch's FashionMNIST dataset? The FashionMNIST dataset has 10 different output classes. How can I get a subset of this dataset with only specific classes? In my case, I only want images of sneaker, pullover, sandal and shirt classes (their classes are 7,2,5 and 6 respectively). This is how I load my dataset. train_dataset_full = torchvision.datasets.FashionMNIST(data_folder, train = True, download = True, transform = transforms.ToTensor()) The approach I’ve followed is below. Iterate through the dataset, one by one, then compare the 1st element (i.e. class) in the returned tuple to my required class. I’m stuck here. If the value returned is true, how can I append/add this observation to an empty dataset?
```
sneaker = 0
pullover = 0
sandal = 0
shirt = 0
for i in range(60000):
    if train_dataset_full[i][1] == 7:
        sneaker += 1
    elif train_dataset_full[i][1] == 2:
        pullover += 1
    elif train_dataset_full[i][1] == 5:
        sandal += 1
    elif train_dataset_full[i][1] == 6:
        shirt += 1
```
Now, in place of sneaker += 1, pullover += 1, sandal += 1 and shirt += 1 I want to do something like this empty_dataset.append(train_dataset_full[i]) or something similar. If the above approach is incorrect, please suggest another method.

Response:
Finally found the answer.
```
dataset_full = torchvision.datasets.FashionMNIST(data_folder, train = True, download = True, transform = transforms.ToTensor())
# Selecting classes 7, 2, 5 and 6
idx = (dataset_full.targets==7) | (dataset_full.targets==2) | (dataset_full.targets==5) | (dataset_full.targets==6)
dataset_full.targets = dataset_full.targets[idx]
dataset_full.data = dataset_full.data[idx]
```


Question:
FastAi What does the slice(lr) do in fit_one_cycle() In Lesson 3 - planet, I saw these 2 lines of code:
```
lr = 0.01
learn.fit_one_cycle(5, slice(lr))
```
if the slice(min_lr, max_lr) then I understand the fit_one_cycle() will use the spread-out Learning Rates from slice(min_lr, max_lr). (Hopefully, my understanding to this is correct) But in this case slice(lr) only has one parameter, What are the differences between fit_one_cycle(5, lr) and fit_one_cycle(5, slice(lr)) ? And what are the benefits of using slice(lr) instead of lr directly?

Response: Jeremy took a while to explain what slice does in Lesson 5. What I understood was that the fastai.vision module divides the architecture in 3 groups and trains them with variable learning rates depending on what you input. (Starting layers usually don't require large variations in parameters) Additionally, if you use 'fit_one_cycle', all the groups will have learning rate annealing with their respective variable learning. Check Lesson 5 https://course.fast.ai/videos/?lesson=5 (use the transcript finder to quickly go to the 'slice' part)


Question: constants in Pytorch Linear Module Class Definition What is __constants__ in pytorch class Linear(Module): defined in https://pytorch.org/docs/stable/_modules/torch/nn/modules/linear.html? What is its functionality and why is it used? I have been searching around, but did not find any documentation. Please note that this does not mean the __constants__ in torch script.

Response:
The __constants__ you're talking about is, in fact, the one related to TorchScript. You can confirm it by using git blame (when it was added and by who) on GitHub. For example, for torch/nn/modules/linear.py, check its git blame. TorchScript also provides a way to use constants that are defined in Python. These can be used to hard-code hyper-parameters into the function, or to define universal constants. -- Attributes of a ScriptModule can be marked constant by listing them as a member of the constants property of the class:
```py
class Foo(torch.jit.ScriptModule):
    __constants__ = ['a']

    def __init__(self):
        super(Foo, self).__init__(False)
        self.a = 1 + 4

   @torch.jit.script_method
   def forward(self, input):
       return self.a + input
```


Question: PyTorch ROCm is out - How to select Radeon GPU as device since Pytorch released the ROCm version, which enables me to use other gpus than nvidias, how can I select my radeon gpu as device in python? Obviously, code like device = torch.cuda.is_available or device = torch.device("cuda") is not working. Thanks for any help. :)

Response: This took me forever to figure out. I'm still having some configuration issues with my AMD GPU, so I haven't been able to test that this works, but, according to this github pytorch thread, the Rocm integration is written so you can just call torch.device('cuda') and no actual porting is required! See the thread here: https://github.com/pytorch/pytorch/issues/10670


Question:
Is SGD optimizer in PyTorch actually does Gradient Descent algorithm? I'm working on trying to compare the converge rate of SGD and GD algorithms for the neural networks. In PyTorch, we often use SGD optimizer as follows.
```
train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=64, shuffle=True)
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
```
```
for epoch in range(epochs):
    running_loss = 0

    for input_batch, labels_batch in train_dataloader:

        input = input_batch

        y_hat = model(input)
        y = labels_batch
        L = loss(y_hat, y)

        optimizer.zero_grad()
        L.backward()

        optimizer.step()

        running_loss += L.item()
```
My understanding about the optimizer here is that the SGD optimizer actually does the Mini-batch Gradient Descent algorithm because we feed the optimizer one batch of data at one time. So, if we set the batch_size parameter as the size of all data, the code actually does Gradient Descent for the neural network. Is my understanding correct?

Response: Your understanding is correct. SGD is just updating weights based on the gradient computed by backpropagation. The flavor of gradient descent that it performs is therefore determined by the data loader. Gradient descent (aka batch gradient descent): Batch size equal to the size of the entire training dataset. Stochastic gradient descent: Batch size equal to one and shuffle=True. Mini-batch gradient descent: Any other batch size and shuffle=True. By far the most common in practical applications.


Question:
BertTokenizer - when encoding and decoding sequences extra spaces appear When using Transformers from HuggingFace I am facing a problem with the encoding and decoding method. I have a the following string:
```
test_string = 'text with percentage%'
```
Then I am running the following code:
```
import torch
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')

test_string = 'text with percentage%'

# encode Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary.
input_ids = tokenizer.encode(test_string)
output = tokenizer.decode(input_ids)
```
And the output looks like this:
```
'text with percentage %'
```
With an extra space before the %. I have tried the extra arguments like clean_up_tokenization_spaces but this is for something different. How what should I use in the decoding and encoding to get exactly the same text before and after. This also happens for other special signs.

Response:
If you are trying to use BERT for token classification in order to find a span in your original string, then one workaround is to use BertTokenizerFast with the option return_offsets_mapping=True.
```
test_string = 'text with percentage%'

tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')
tokens = tokenizer(test_string, return_offsets_mapping=True)
input_ids = tokens.data["input_ids"]

span_start_index, span_stop_index = some_model(input_ids)
```
Then once you get the token classification results, you can do something like
```
predicted_span = test_string[tokens.encodings[0].offsets[span_start_index][0]:tokens.encodings[0].offsets[span_stop_index][1]]
```


Question:
Correct way to use custom weight maps in unet architecture There is a famous trick in u-net architecture to use custom weight maps to increase accuracy. Below are the details of it: Now, by asking here and at multiple other place, I get to know about 2 approaches. I want to know which one is correct or is there any other right approach which is more correct? First is to use torch.nn.Functional method in the training loop: loss = torch.nn.functional.cross_entropy(output, target, w) where w will be the calculated custom weight. Second is to use reduction='none' in the calling of loss function outside the training loop criterion = torch.nn.CrossEntropy(reduction='none') and then in the training loop multiplying with the custom weight:
```
gt # Ground truth, format torch.long
pd # Network output
W # per-element weighting based on the distance map from UNet
loss = criterion(pd, gt)
loss = W*loss # Ensure that weights are scaled appropriately
loss = torch.sum(loss.flatten(start_dim=1), axis=0) # Sums the loss per image
loss = torch.mean(loss) # Average across a batch
```
Now, I am kinda confused which one is right or is there any other way, or both are right?

Response:
The weighting portion looks like just simply weighted cross entropy which is performed like this for the number of classes (2 in the example below).
```
weights = torch.FloatTensor([.3, .7])
loss_func = nn.CrossEntropyLoss(weight=weights)
```
EDIT: Have you seen this implementation from Patrick Black?
```
# Set properties
batch_size = 10
out_channels = 2
W = 10
H = 10

# Initialize logits etc. with random
logits = torch.FloatTensor(batch_size, out_channels, H, W).normal_()
target = torch.LongTensor(batch_size, H, W).random_(0, out_channels)
weights = torch.FloatTensor(batch_size, 1, H, W).random_(1, 3)

# Calculate log probabilities
logp = F.log_softmax(logits)

# Gather log probabilities with respect to target
logp = logp.gather(1, target.view(batch_size, 1, H, W))

# Multiply with weights
weighted_logp = (logp * weights).view(batch_size, -1)

# Rescale so that loss is in approx. same interval
weighted_loss = weighted_logp.sum(1) / weights.view(batch_size, -1).sum(1)

# Average over mini-batch
weighted_loss = -1. * weighted_loss.mean()
```


Question:
Batch-Matrix multiplication in Pytorch - Confused with the handling of the output's dimension I got two arrays :
```
A
B
```
Array A contains a batch of RGB images, with shape:
```
[batch, Width, Height, 3]
```
whereas Array B contains coefficients needed for a "transformation-like" operation on images, with shape:
```
[batch, 4, 4, 3]
```
To put it simply, the operation for a single image is a multiplication that outputs an environment map (normalMap * Coefficients). The output I want should hold shape:
```
[batch, Width, Height, 3]
```
I tried using torch.bmm but failed. Is this possible somehow?

Response:
I think you need to calculate that PyTorch works with
```
BxCxHxW : number of mini-batches, channels, height, width
```
format, and also use matmul, since bmm works with tensors or ndim/dim/rank =3. I know you may find this online, but for any case:
```
batch1 = torch.randn(10, 3, 20, 10)
batch2 = torch.randn(10, 3, 10, 30)
res = torch.matmul(batch1, batch2)
res.size() # torch.Size([10, 3, 20, 30])
```


Question:
How `backward()` works in PyTorch? I recently study PyTorch and backward(). I understood how to use it, but when I try:
```
x = Variable(2*torch.ones(2, 2), requires_grad=True)
x.backward(x)
print(x.grad)
```
I expect:
```
tensor([[1., 1.],
        [1., 1.]])
```
because it is an identity function. However, it returns:
```
tensor([[2., 2.],
        [2., 2.]]).
```
Why this happens?

Response:
Actually, this is what you are looking for: Case 1: when z = 2*x**3 + x
```
import torch
from torch.autograd import Variable
x = Variable(2*torch.ones(2, 2), requires_grad=True)
z = x*x*x*2+x
z.backward(torch.ones_like(z))
print(x.grad)
```
output:
```
tensor([[25., 25.],
        [25., 25.]])
```
Case 2: when z = x*x
```
x = Variable(2*torch.ones(2, 2), requires_grad=True)
z = x*x
z.backward(torch.ones_like(z))
print(x.grad)
```
output:
```
tensor([[4., 4.],
        [4., 4.]])
```
Case 3: when z = x (your case)
```
x = Variable(2*torch.ones(2, 2), requires_grad=True)
z = x
z.backward(torch.ones_like(z))
print(x.grad)
```
output:
```
tensor([[1., 1.],
        [1., 1.]])
```
To learn more how to calculate gradient in pytorch, check this.


Question:
How to share a list of tensors in PyTorch multiprocessing? I am programming with PyTorch multiprocessing. I want all the subprocesses can read/write the same list of tensors (no resize). For example the variable can be
```
m = list(torch.randn(3), torch.randn(5))
```
Because each tensor has different sizes, I cannot organize them into a single tensor. A python list has no share_memory_() function, and multiprocessing.Manager cannot handle a list of tensors. How can I share the variable m among multiple subprocesses?

Response:
I found the solution by myself. It is pretty straightforward. Just call share_memory_() for each list elements. The list itself is not in the shared memory, but the list elements are. Demo code
```py
import torch.multiprocessing as mp
import torch

def foo(worker,tl):
    tl[worker] += (worker+1) * 1000

if __name__ == '__main__':
    tl = [torch.randn(2), torch.randn(3)]

    for t in tl:
        t.share_memory_()

    print("before mp: tl=")
    print(tl)

    p0 = mp.Process(target=foo, args=(0, tl))
    p1 = mp.Process(target=foo, args=(1, tl))
    p0.start()
    p1.start()
    p0.join()
    p1.join()

    print("after mp: tl=")
    print(tl)
```
Output
```
before mp: tl=
[
 1.5999
 2.2733
[torch.FloatTensor of size 2]
,
 0.0586
 0.6377
-0.9631
[torch.FloatTensor of size 3]
]

after mp: tl=
[
 1001.5999
 1002.2733
[torch.FloatTensor of size 2]
,
 2000.0586
 2000.6377
 1999.0370
[torch.FloatTensor of size 3]
]
```


Question: What does model.eval() do in pytorch? When should I use .eval()? I understand it is supposed to allow me to "evaluate my model". How do I turn it back off for training? Example training code using .eval().

Response:
model.eval() is a kind of switch for some specific layers/parts of the model that behave differently during training and inference (evaluating) time. For example, Dropouts Layers, BatchNorm Layers etc. You need to turn them off during model evaluation, and .eval() will do it for you. In addition, the common practice for evaluating/validation is using torch.no_grad() in pair with model.eval() to turn off gradients computation:
```py
# evaluate model:
model.eval()

with torch.no_grad():
    ...
    out_data = model(data)
    ...
```
BUT, don't forget to turn back to training mode after eval step:
```py
# training step
...
model.train()
...
```


Question:
Converting a scipy coo_matrix to pytorch sparse tensor I have a coo_matrix:
```
from scipy.sparse import coo_matrix
coo = coo_matrix((3, 4), dtype = "int8")
```
That I want converted to a pytorch sparse tensor. According to the documentation https://pytorch.org/docs/master/sparse.html it should follow the coo format, but I cannot find a simple way to do the conversion. Any help would be greatly appreciated!

Response:
Using the data as in the Pytorch docs, it can be done simply using the attributes of the Numpy coo_matrix:
```
import torch
import numpy as np
from scipy.sparse import coo_matrix

coo = coo_matrix(([3,4,5], ([0,1,1], [2,0,2])), shape=(2,3))

values = coo.data
indices = np.vstack((coo.row, coo.col))

i = torch.LongTensor(indices)
v = torch.FloatTensor(values)
shape = coo.shape

torch.sparse.FloatTensor(i, v, torch.Size(shape)).to_dense()
```
Output
```
0 0 3
4 0 5
[torch.FloatTensor of size 2x3]
```


Question: what does padding_idx do in nn.embeddings() I'm learning pytorch and I'm wondering what does the padding_idx attribute do in torch.nn.Embedding(n1, d1, padding_idx=0)? I have looked everywhere and couldn't find something I can get. Can you show example to illustrate this?

Response:
padding_idx is indeed quite badly described in the documentation. Basically, it specifies which index passed during call will mean "zero vector" (which is quite often used in NLP in case some token is missing). By default no index will mean "zero vector", as you can see in the example below:
```
import torch

embedding = torch.nn.Embedding(10, 3)
input = torch.LongTensor([[0, 1, 0, 5]])
print(embedding(input))
```
Will give you:
```
tensor([[[ 0.1280, -1.1390, -2.5007],
         [ 0.3617, -0.9280,  1.2894],
         [ 0.1280, -1.1390, -2.5007],
         [-1.3135, -0.0229,  0.2451]]], grad_fn=<EmbeddingBackward>)
```
If you specify padding_idx=0 every input where the value is equal to 0 (so zero-th and second row) will be zero-ed out like this (code: embedding = torch.nn.Embedding(10, 3, padding_idx=0)):
```
tensor([[[ 0.0000,  0.0000,  0.0000],
         [-0.4448, -0.2076,  1.1575],
         [ 0.0000,  0.0000,  0.0000],
         [ 1.3602, -0.6299, -0.5809]]], grad_fn=<EmbeddingBackward>
```
If you were to specify padding_idx=5 last row would be full of zeros etc.


Question: Why do we "pack" the sequences in PyTorch? I was trying to replicate How to use packing for variable-length sequence inputs for rnn but I guess I first need to understand why we need to "pack" the sequence. I understand why we "pad" them but why is "packing" (via pack_padded_sequence) necessary?

Response:
Here are some visual explanations1 that might help to develop better intuition for the functionality of pack_padded_sequence(). TL;DR: It is performed primarily to save compute. Consequently, the time required for training neural network models is also (drastically) reduced, especially when carried out on very large (a.k.a. web-scale) datasets. Let's assume we have 6 sequences (of variable lengths) in total. You can also consider this number 6 as the batch_size hyperparameter. (The batch_size will vary depending on the length of the sequence (cf. Fig.2 below)) Now, we want to pass these sequences to some recurrent neural network architecture(s). To do so, we have to pad all of the sequences (typically with 0s) in our batch to the maximum sequence length in our batch (max(sequence_lengths)), which in the below figure is 9. So, the data preparation work should be complete by now, right? Not really.. Because there is still one pressing problem, mainly in terms of how much compute do we have to do when compared to the actually required computations. For the sake of understanding, let's also assume that we will matrix multiply the above padded_batch_of_sequences of shape (6, 9) with a weight matrix W of shape (9, 3). Thus, we will have to perform 6x9 = 54 multiplication and 6x8 = 48 addition (nrows x (n-1)_cols) operations, only to throw away most of the computed results since they would be 0s (where we have pads). The actual required compute in this case is as follows:
```
9-mult  8-add
 8-mult  7-add
 6-mult  5-add
 4-mult  3-add
 3-mult  2-add
 2-mult  1-add
---------------
32-mult  26-add

------------------------------
#savings: 22-mult & 22-add ops
          (32-54)  (26-48)
```
That's a LOT more savings even for this very simple (toy) example. You can now imagine how much compute (eventually: cost, energy, time, carbon emission etc.) can be saved using pack_padded_sequence() for large tensors with millions of entries, and million+ systems all over the world doing that, again and again. The functionality of pack_padded_sequence() can be understood from the figure below, with the help of the used color-coding: As a result of using pack_padded_sequence(), we will get a tuple of tensors containing (i) the flattened (along axis-1, in the above figure) sequences , (ii) the corresponding batch sizes, tensor([6,6,5,4,3,3,2,2,1]) for the above example. The data tensor (i.e. the flattened sequences) could then be passed to objective functions such as CrossEntropy for loss calculations. 1 image credits to @sgrvinod


Question: What does model.train() do in PyTorch? Does it call forward() in nn.Module? I thought when we call the model, forward method is being used. Why do we need to specify train()?

Response: model.train() tells your model that you are training the model. This helps inform layers such as Dropout and BatchNorm, which are designed to behave differently during training and evaluation. For instance, in training mode, BatchNorm updates a moving average on each new batch; whereas, for evaluation mode, these updates are frozen. More details: model.train() sets the mode to train (see source code). You can call either model.eval() or model.train(mode=False) to tell that you are testing. It is somewhat intuitive to expect train function to train model but it does not do that. It just sets the mode.


Question: Torchtext 0.7 shows Field is being deprecated. What is the alternative? Looks like the previous paradigm of declaring Fields, Examples and using BucketIterator is deprecated and will move to legacy in 0.8. However, I don't seem to be able to find an example of the new paradigm for custom datasets (as in, not the ones included in torch.datasets) that doesn't use Field. Can anyone point me at an up-to-date example? Reference for deprecation: https://github.com/pytorch/text/releases

Response:
It took me a little while to find the solution myself. The new paradigm is like so for prebuilt datasets:
```
from torchtext.experimental.datasets import AG_NEWS
train, test = AG_NEWS(ngrams=3)
```
or like so for custom built datasets:
```
from torch.utils.data import DataLoader
def collate_fn(batch):
    texts, labels = [], []
    for label, txt in batch:
        texts.append(txt)
        labels.append(label)
    return texts, labels
dataloader = DataLoader(train, batch_size=8, collate_fn=collate_fn)
for idx, (texts, labels) in enumerate(dataloader):
    print(idx, texts, labels)
```
I've copied the examples from the Source


Question: Printing all the contents of a tensor I came across this PyTorch tutorial (in neural_networks_tutorial.py) where they construct a simple neural network and run an inference. I would like to print the contents of the entire input tensor for debugging purposes. What I get when I try to print the tensor is something like this and not the entire tensor: I saw a similar link for numpy but was not sure about what would work for PyTorch. I can convert it to numpy and may be view it, but would like to avoid the extra overhead. Is there a way for me to print the entire tensor?

Response:
To avoid truncation and to control how much of the tensor data is printed use the same API as numpy's numpy.set_printoptions(threshold=10_000). Example:
```
x = torch.rand(1000, 2, 2)
print(x) # prints the truncated tensor
torch.set_printoptions(threshold=10_000)
print(x) # prints the whole tensor
```
If your tensor is very large, adjust the threshold value to a higher number. Another option is:
```
torch.set_printoptions(profile="full")
print(x) # prints the whole tensor
torch.set_printoptions(profile="default") # reset
print(x) # prints the truncated tensor
```
All the available set_printoptions arguments are documented here.


Question:
Pytorch: RuntimeError: result type Float can't be cast to the desired output type Long I have a model which looks as follows:
```py
IMG_WIDTH = IMG_HEIGHT = 224

class AlexNet(nn.Module):
  def __init__(self, output_dim):
    super(AlexNet, self).__init__()
    self._to_linear = None
    self.x = torch.randn(3, IMG_WIDTH, IMG_HEIGHT).view(-1, 3, IMG_WIDTH, IMG_HEIGHT)
    self.features = nn.Sequential(
        nn.Conv2d(3, 64, 3, 2, 1), # in_channels, out_channels, kernel_size, stride, padding
        nn.MaxPool2d(2),
        nn.ReLU(inplace=True),
        nn.Conv2d(64, 192, 3, padding=1),
        nn.MaxPool2d(2),
        nn.ReLU(inplace=True),
        nn.Conv2d(192, 384, 3, padding=1),
        nn.MaxPool2d(2),
        nn.ReLU(inplace=True),
        nn.Conv2d(384, 256, 3, padding=1),
        nn.MaxPool2d(2),
        nn.ReLU(inplace=True),
        nn.Conv2d(256, 512, 3, padding=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(512, 256, 3, padding=1),
        nn.MaxPool2d(2),
        nn.ReLU(inplace=True)
  )
    self.conv(self.x)
    self.classifier = nn.Sequential(
        nn.Dropout(.5),
        nn.Linear(self._to_linear, 4096),
        nn.ReLU(inplace=True),
        nn.Dropout(.5),
        nn.Linear(4096, 4096),
        nn.ReLU(inplace=True),
        nn.Linear(4096, output_dim),
    )

  def conv(self, x):
    x = self.features(x)
    if self._to_linear is None:
        self._to_linear = x.shape[1] * x.shape[2] * x.shape[3]
    return x

  def forward(self, x):
    x = self.conv(x)
    h = x.view(x.shape[0], -1)
    x = self.classifier(h)
    return x, h
```
Here is my optimizer and loss functions:
```py
optimizer = torch.optim.Adam(model.parameters())
criterion = nn.BCEWithLogitsLoss().to(device)
```
Here is my train and evaluate functions:
```py
def train(model, iterator, optimizer, criterion, device):
  epoch_loss, epoch_acc = 0, 0
  model.train()
  for (x, y) in iterator:
    # features and labels to the device
    x = x.to(device)
    y = y.to(device).long()
    # Zero the gradients
    optimizer.zero_grad()
    y_pred, _ = model(x)

    # Calculate the loss and accuracy
    loss = criterion(y_pred.squeeze(), y)
    acc = binary_accuracy(y_pred, y)
    # Backward propagate
    loss.backward()
    # Update the weights
    optimizer.step()

    epoch_loss +=loss.item()
    epoch_acc += acc.item()

  return epoch_loss/len(iterator), epoch_acc/len(iterator)

def evaluate(model, iterator, criterion, device):
  epoch_loss, epoch_acc = 0, 0
  model.eval()
  with torch.no_grad():
    for (x, y) in iterator:
      x = x.to(device)
      y = y.to(device).long()
      y_pred, _ = model(x)
      loss = criterion(y_pred, y)
      acc = binary_accuracy(y_pred, y)

      epoch_loss += loss.item()
      epoch_acc += acc.item()
  return epoch_loss/len(iterator), epoch_acc/len(iterator)
```
This is the error that I'm getting:
```
RuntimeError: result type Float can't be cast to the desired output type Long
```
What may be possibly my problem because I have tried to convert my labels to long tensors as follows:
```
y = y.to(device).long()
```
But it seems not to work.

Response:
I was getting the same error doing this:
```
loss_fn(output, target)
```
where the output was Tensor torch.float32 and target was Tensor torch.int64. What solved this problem was calling the loss function like this:
```
loss_fn(output, target.float())
```


Question: PyTorch Optimizer: AdamW and Adam with weight decay Is there any difference between torch.optim.Adam(weight_decay=0.01) and torch.optim.AdamW(weight_decay=0.01)? Link to the docs: torch.optim.

Response:
Yes, Adam and AdamW weight decay are different. Hutter pointed out in their paper (Decoupled Weight Decay Regularization) that the way weight decay is implemented in Adam in every library seems to be wrong, and proposed a simple way (which they call AdamW) to fix it. In Adam, the weight decay is usually implemented by adding wd*w (wd is weight decay here) to the gradients (Ist case), rather than actually subtracting from weights (IInd case).
```
# Ist: Adam weight decay implementation (L2 regularization)
final_loss = loss + wd * all_weights.pow(2).sum() / 2
# IInd: equivalent to this in SGD
w = w - lr * w.grad - lr * wd * w
```
These methods are same for vanilla SGD, but as soon as we add momentum, or use a more sophisticated optimizer like Adam, L2 regularization (first equation) and weight decay (second equation) become different. AdamW follows the second equation for weight decay. In Adam weight_decay (float, optional) – weight decay (L2 penalty) (default: 0) In AdamW weight_decay (float, optional) – weight decay coefficient (default: 1e-2) Read more on the fastai blog.


Question:
How can I invert a MelSpectrogram with torchaudio and get an audio waveform? I have a MelSpectrogram generated from:
```
eval_seq_specgram = torchaudio.transforms.MelSpectrogram(sample_rate=sample_rate, n_fft=256)(eval_audio_data).transpose(1, 2)
```
So eval_seq_specgram now has a size of torch.Size([1, 128, 499]), where 499 is the number of timesteps and 128 is the n_mels. I'm trying to invert it, so I'm trying to use GriffinLim, but before doing that, I think I need to invert the melscale, so I have:
```
inverse_mel_pred = torchaudio.transforms.InverseMelScale(sample_rate=sample_rate, n_stft=256)(eval_seq_specgram)
```
inverse_mel_pred has a size of torch.Size([1, 256, 499]) Then I'm trying to use GriffinLim:
```
pred_audio = torchaudio.transforms.GriffinLim(n_fft=256)(inverse_mel_pred)
```
but I get an error:
```
Traceback (most recent call last):
  File "evaluate_spect.py", line 63, in <module>
    main()
  File "evaluate_spect.py", line 51, in main
    pred_audio = torchaudio.transforms.GriffinLim(n_fft=256)(inverse_mel_pred)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torchaudio/transforms.py", line 169, in forward
    return F.griffinlim(specgram, self.window, self.n_fft, self.hop_length, self.win_length, self.power,
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torchaudio/functional.py", line 179, in griffinlim
    inverse = torch.istft(specgram * angles,
RuntimeError: The size of tensor a (256) must match the size of tensor b (129) at non-singleton dimension 1
```
Not sure what I'm doing wrong or how to resolve this.

Response:
Just for history, full code:
```
import torch
import torchaudio
import IPython

waveform, sample_rate = torchaudio.load("wavs/LJ030-0196.wav", normalize=True)

n_fft = 256

n_stft = int((n_fft//2) + 1)
transofrm = torchaudio.transforms.MelSpectrogram(sample_rate, n_fft=n_fft)
invers_transform = torchaudio.transforms.InverseMelScale(sample_rate=sample_rate, n_stft=n_stft)
grifflim_transform = torchaudio.transforms.GriffinLim(n_fft=n_fft)

mel_specgram = transofrm(waveform)
inverse_waveform = invers_transform(mel_specgram)
pseudo_waveform = grifflim_transform(inverse_waveform)
```
And
```
IPython.display.Audio(waveform.numpy(), rate=sample_rate)
```
```
IPython.display.Audio(pseudo_waveform.numpy(), rate=sample_rate)
```


Question: PyTorch flatten doesn't maintain batch size In Keras, using the Flatten() layer retains the batch size. For eg, if the input shape to Flatten is (32, 100, 100), in Keras output of Flatten is (32, 10000), but in PyTorch it is 320000. Why is it so?

Response: As OP already pointed out in their answer, the tensor operations do not default to considering a batch dimension. You can use torch.flatten() or Tensor.flatten() with start_dim=1 to start the flattening operation after the batch dimension. Alternatively since PyTorch 1.2.0 you can define an nn.Flatten() layer in your model which defaults to start_dim=1.


Question: KL Divergence for two probability distributions in PyTorch I have two probability distributions. How should I find the KL-divergence between them in PyTorch? The regular cross entropy only accepts integer labels.

Response:
Yes, PyTorch has a method named kl_div under torch.nn.functional to directly compute KL-devergence between tensors. Suppose you have tensor a and b of same shape. You can use the following code:
```
import torch.nn.functional as F
out = F.kl_div(a, b)
```
For more details, see the above method documentation.