Question:
Given input size: (128x1x1). Calculated output size: (128x0x0). Output size is too small I am trying to train a U-Net which looks like this
```
`class UNet(nn.Module):
def __init__(self, imsize):
    super(UNet, self).__init__()
    self.imsize = imsize

    self.activation = F.relu
    self.pool1 = nn.MaxPool2d(2)
    self.pool2 = nn.MaxPool2d(2)
    self.pool3 = nn.MaxPool2d(2)
    self.pool4 = nn.MaxPool2d(2)
    self.conv_block1_64 = UNetConvBlock(4, 64)
    self.conv_block64_128 = UNetConvBlock(64, 128)
    self.conv_block128_256 = UNetConvBlock(128, 256)
    self.conv_block256_512 = UNetConvBlock(256, 512)
    self.conv_block512_1024 = UNetConvBlock(512, 1024)

    self.up_block1024_512 = UNetUpBlock(1024, 512)
    self.up_block512_256 = UNetUpBlock(512, 256)
    self.up_block256_128 = UNetUpBlock(256, 128)
    self.up_block128_64 = UNetUpBlock(128, 64)

    self.last = nn.Conv2d(64, 1, 1)`
```
The loss function i am using is
```
`class BCELoss2d(nn.Module):

def __init__(self, weight=None, size_average=True):
    super(BCELoss2d, self).__init__()
    self.bce_loss = nn.BCELoss(weight, size_average)

def forward(self, logits, targets):
    probs = F.sigmoid(logits)
    probs_flat = probs.view(-1)
    targets_flat = targets.view(-1)
    return self.bce_loss(probs_flat, targets_flat)`
```
The input image tensor is [1,1,68,68] and labels are also of the same shape I get this error:
```
<ipython-input-72-270210759010> in forward(self, x)
 75
 76         block4 = self.conv_block256_512(pool3)
---> 77         pool4 = self.pool4(block4)
     78
  79         block5 = self.conv_block512_1024(pool4)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py in _    _call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/pooling.py in forward(self, input)
    141         return F.max_pool2d(input, self.kernel_size, self.stride,
    142                             self.padding, self.dilation, self.ceil_mode,
--> 143                             self.return_indices)
    144
    145     def __repr__(self):

/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py in max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode, return_indices)
    332     See :class:`~torch.nn.MaxPool2d` for details.
    333     """
--> 334     ret = torch._C._nn.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
    335     return ret if return_indices else ret[0]
    336

RuntimeError: Given input size: (128x1x1). Calculated output size: (128x0x0). Output size is too small at /pytorch/torch/lib/THCUNN/generic/SpatialDilatedMaxPooling.cu:69
```
I'm guessing I'm making a mistake in my channel size or pooling size but i'm not sure where exactly is the mistake.

Response: Your problem is that before the Pool4 your image has already reduced to a 1x1pixel size image. So you need to either feed an much larger image of size at least around double that (~134x134) or remove a pooling layer in your network.


Question:
PyTorch : How to apply the same random transformation to multiple image? I am writing a simple transformation for a dataset which contains many pairs of images. As a data augmentation, I want to apply some random transformation for each pair but the images in that pair should be transformed in the same way. For example, given a pair of two images A and B, if A is flipped horizontally, B must be flipped horizontally as A. Then the next pair C and D should be differently transformed from A and B but C and D are transformed in the same way. I am trying that in the way below
```py
import random
import numpy as np
import torchvision.transforms as transforms
from PIL import Image

img_a = Image.open("sample_ajpg") # note that two images have the same size
img_b = Image.open("sample_b.png")
img_c, img_d = Image.open("sample_c.jpg"), Image.open("sample_d.png")

transform = transforms.RandomChoice(
    [transforms.RandomHorizontalFlip(),
     transforms.RandomVerticalFlip()]
)
random.seed(0)
display(transform(img_a))
display(transform(img_b))

random.seed(1)
display(transform(img_c))
display(transform(img_d))
```
Yet、 the above code does not choose the same transformation and as I tested, it is dependent on the number of times transform is called. Is there any way to force transforms.RandomChoice to use the same transform when specified?

Response:
Usually a workaround is to apply the transform on the first image, retrieve the parameters of that transform, then apply with a deterministic transform with those parameters on the remaining images. However, here RandomChoice does not provide an API to get the parameters of the applied transform since it involves a variable number of transforms. In those cases, I usually implement an overwrite to the original function. Looking at the torchvision implementation, it's as simple as:
```
class RandomChoice(RandomTransforms):
    def __call__(self, img):
        t = random.choice(self.transforms)
        return t(img)
```
Here are two possible solutions. You can either sample from the transform list on __init__ instead of on __call__:
```
import random
import torchvision.transforms as T

class RandomChoice(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.t = random.choice(self.transforms)

    def __call__(self, img):
        return self.t(img)
```
So you can do:
```
transform = RandomChoice([
     T.RandomHorizontalFlip(),
     T.RandomVerticalFlip()
])
display(transform(img_a)) # both img_a and img_b will
display(transform(img_b)) # have the same transform

transform = RandomChoice([
    T.RandomHorizontalFlip(),
    T.RandomVerticalFlip()
])
display(transform(img_c)) # both img_c and img_d will
display(transform(img_d)) # have the same transform
```
Or better yet, transform the images in batch:
```
import random
import torchvision.transforms as T

class RandomChoice(torch.nn.Module):
    def __init__(self, transforms):
       super().__init__()
       self.transforms = transforms

    def __call__(self, imgs):
        t = random.choice(self.transforms)
        return [t(img) for img in imgs]
```
Which allows to do:
```
transform = RandomChoice([
     T.RandomHorizontalFlip(),
     T.RandomVerticalFlip()
])

img_at, img_bt = transform([img_a, img_b])
display(img_at) # both img_a and img_b will
display(img_bt) # have the same transform

img_ct, img_dt = transform([img_c, img_d])
display(img_ct) # both img_c and img_d will
display(img_dt) # have the same transform
```


Question:
PyTorch Lightning move tensor to correct device in validation_epoch_end I would like to create a new tensor in a validation_epoch_end method of a LightningModule. From the official docs (page 48) it is stated that we should avoid direct .cuda() or .to(device) calls: There are no .cuda() or .to() calls. . . Lightning does these for you. and we are encouraged to use type_as method to transfer to the correct device. new_x = new_x.type_as(x.type()) However, in a step validation_epoch_end I do not have any tensor to copy device from(by type_as method) in a clean way. My question is what should I do if I want to create a new tensor in this method and transfer it to the device where is the model? The only thing I can think of is to find a tensor in the outputs dictionary but it feels kinda messy:
```
avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
output = self(self.__test_input.type_as(avg_loss))
```
Is there any clean way to achieve that?

Response:
did you check part 3.4 (page 34) in the doc you linked ? LightningModules know what device they are on! construct tensors on the device directly to avoid CPU->Device transfer
```
t = tensor.rand(2, 2).cuda()# bad
(self is lightningModule)t = tensor.rand(2,2, device=self.device)# good
```
I had a similar issue to create tensors this helped me. I hope it will help you too.


Question: What is the difference between 'torch.backends.cudnn.deterministic=True' and 'torch.set_deterministic(True)'? My network includes torch.nn.MaxPool3d which throw a RuntimeError when the cudnn deterministic flag is on according to the PyTorch docs (version 1.7 - https://pytorch.org/docs/stable/generated/torch.set_deterministic.html#torch.set_deterministic). However, when I inserted the code torch.backends.cudnn.deterministic=True at the beginning of my code, there was no RuntimeError. Why doesn't that code throw a RuntimeError? I wonder whether that code guarantees the deterministic computation of my training process.

Response: torch.backends.cudnn.deterministic=True only applies to CUDA convolution operations, and nothing else. Therefore, no, it will not guarantee that your training process is deterministic, since you're also using torch.nn.MaxPool3d, whose backward function is nondeterministic for CUDA. torch.set_deterministic(), on the other hand, affects all the normally-nondeterministic operations listed here (note that set_deterministic has been renamed to use_deterministic_algorithms in 1.8): https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html?highlight=use_deterministic#torch.use_deterministic_algorithms As the documentation states, some of the listed operations don't have a deterministic implementation. So if torch.use_deterministic_algorithms(True) is set, they will throw an error. If you need to use nondeterministic operations like torch.nn.MaxPool3d, then, at the moment, there is no way for your training process to be deterministic--unless you write a custom deterministic implementation yourself. Or you could open a GitHub issue requesting a deterministic implementation: https://github.com/pytorch/pytorch/issues In addition, you might want to check out this page: https://pytorch.org/docs/stable/notes/randomness.html


Question:
Difference between Keras' BatchNormalization and PyTorch's BatchNorm2d? I've a sample tiny CNN implemented in both Keras and PyTorch. When I print summary of both the networks, the total number of trainable parameters are same but total number of parameters and number of parameters for Batch Normalization don't match. Here is the CNN implementation in Keras:
```
inputs = Input(shape = (64, 64, 1)). # Channel Last: (NHWC)

model = Conv2D(filters=32, kernel_size=(3, 3), padding='SAME', activation='relu', input_shape=(IMG_SIZE, IMG_SIZE, 1))(inputs)
model = BatchNormalization(momentum=0.15, axis=-1)(model)
model = Flatten()(model)

dense = Dense(100, activation = "relu")(model)
head_root = Dense(10, activation = 'softmax')(dense)
```
And the summary printed for above model is:
```
Model: "model_8"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_9 (InputLayer)         (None, 64, 64, 1)         0
_________________________________________________________________
conv2d_10 (Conv2D)           (None, 64, 64, 32)        320
_________________________________________________________________
batch_normalization_2 (Batch (None, 64, 64, 32)        128
_________________________________________________________________
flatten_3 (Flatten)          (None, 131072)            0
_________________________________________________________________
dense_11 (Dense)             (None, 100)               13107300
_________________________________________________________________
dense_12 (Dense)             (None, 10)                1010
=================================================================
Total params: 13,108,758
Trainable params: 13,108,694
Non-trainable params: 64
_________________________________________________________________
```
Here's the implementation of the same model architecture in PyTorch:
```
# Image format: Channel first (NCHW) in PyTorch
class CustomModel(nn.Module):
def __init__(self):
    super(CustomModel, self).__init__()
    self.layer1 = nn.Sequential(
        nn.Conv2d(in_channels=1, out_channels=32, kernel_size=(3, 3), padding=1),
        nn.ReLU(True),
        nn.BatchNorm2d(num_features=32),
    )
    self.flatten = nn.Flatten()
    self.fc1 = nn.Linear(in_features=131072, out_features=100)
    self.fc2 = nn.Linear(in_features=100, out_features=10)

def forward(self, x):
    output = self.layer1(x)
    output = self.flatten(output)
    output = self.fc1(output)
    output = self.fc2(output)
    return output
```
And following is the output of summary of the above model:
```
----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1           [-1, 32, 64, 64]             320
              ReLU-2           [-1, 32, 64, 64]               0
       BatchNorm2d-3           [-1, 32, 64, 64]              64
           Flatten-4               [-1, 131072]               0
            Linear-5                  [-1, 100]      13,107,300
            Linear-6                   [-1, 10]           1,010
================================================================
Total params: 13,108,694
Trainable params: 13,108,694
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.02
Forward/backward pass size (MB): 4.00
Params size (MB): 50.01
Estimated Total Size (MB): 54.02
----------------------------------------------------------------
```
As you can see in above results, Batch Normalization in Keras has more number of parameters than PyTorch (2x to be exact). So what's the difference in above CNN architectures? If they are equivalent, then what am I missing here?

Response: Keras treats as parameters (weights) many things that will be "saved/loaded" in the layer. While both implementations naturally have the accumulated "mean" and "variance" of the batches, these values are not trainable with backpropagation. Nevertheless, these values are updated every batch, and Keras treats them as non-trainable weights, while PyTorch simply hides them. The term "non-trainable" here means "not trainable by backpropagation", but doesn't mean the values are frozen. In total they are 4 groups of "weights" for a BatchNormalization layer. Considering the selected axis (default = -1, size=32 for your layer) scale (32) - trainable offset (32) - trainable accumulated means (32) - non-trainable, but updated every batch accumulated std (32) - non-trainable, but updated every batch The advantage of having it like this in Keras is that when you save the layer, you also save the mean and variance values the same way you save all other weights in the layer automatically. And when you load the layer, these weights are loaded together.


Question:
Why detach needs to be called on variable in this example? I was going through this example - https://github.com/pytorch/examples/blob/master/dcgan/main.py and I have a basic question.
```
fake = netG(noise)
label = Variable(label.fill_(fake_label))
output = netD(fake.detach()) # detach to avoid training G on these labels
errD_fake = criterion(output, label)
errD_fake.backward()
D_G_z1 = output.data.mean()
errD = errD_real + errD_fake
optimizerD.step()
```
I understand that why we call detach() on variable fake, so that no gradients are computed for the Generator parameters. My question is, does it matter since optimizerD.step() is going to update the parameters associated with Discriminator only? OptimizerD is defined as: optimizerD = optim.Adam(netD.parameters(), lr=opt.lr, betas=(opt.beta1, 0.999)) Besides, in the next step when we will update parameters for Generator, before that we will call netG.zero_grad() which eventually removes all previously computed gradients. Moreover, when we update parameters for G network, we do this - output = netD(fake). Here, we are not using detach. Why? So, why detaching the variable (line 3) is necessary in the above code?

Response: ORIGINAL ANSWER (WRONG / INCOMPLETE) You're right, optimizerD only updates netD and the gradients on netG are not used before netG.zero_grad() is called, so detaching is not necessary, it just saves time, because you're not computing gradients for the generator. You're basically also answering your other question yourself, you don't detach fake in the second block because you specifically want to compute gradients on netG to be able to update its parameters. Note how in the second block real_label is used as the corresponding label for fake, so if the discriminator finds the fake input to be real, the final loss is small, and vice versa, which is precisely what you want for the generator. Not sure if that's what confused you, but it's really the only difference compared to training the discriminator on fake inputs. EDIT Please see FatPanda's comment! My original answer is in fact incorrect. Pytorch destroys (parts of) the compute graph when .backward() is called. Without detaching before errD_fake.backward() the errG.backward() call later would not be able to backprop into the generator because the required graph is no longer available (unless you specify retain_graph=True). I'm relieved Soumith made the same mistake :D


Question: Concatenating two tensors with different dimensions in Pytorch Is it possible to concatenate two tensors with different dimensions without using for loop. e.g. Tensor 1 has dimensions (15, 200, 2048) and Tensor 2 has dimensions (1, 200, 2048). Is it possible to concatenate 2nd tensor with 1st tensor along all the 15 indices of 1st dimension in 1st Tensor (Broadcast 2nd tensor along 1st dimension of Tensor 1 while concatenating along 3rd dimension of 1st tensor)? The resulting tensor should have dimensions (15, 200, 4096). Is it possible to accomplish this without for loop ?

Response:
You could do the broadcasting manually (using Tensor.expand()) before the concatenation (using torch.cat()):
```python
import torch

a = torch.randn(15, 200, 2048)
b = torch.randn(1, 200, 2048)

repeat_vals = [a.shape[0] // b.shape[0]] + [-1] * (len(b.shape) - 1)
# or directly repeat_vals = (15, -1, -1) or (15, 200, 2048) if shapes are known and fixed...
res = torch.cat((a, b.expand(*repeat_vals)), dim=-1)
print(res.shape)
# torch.Size([15, 200, 4096])
```


Question:
What's the workaround for "ragged/jagged tensors" in PyTorch? Tensorflow provides ragged tensors (https://www.tensorflow.org/guide/ragged_tensor). PyTorch however doesn't provide such a data structure. Is there a workaround to construct something similar in PyTorch?
```py
import numpy as np
x = np.array([[0], [0, 1]])
print(x)  # [list([0]) list([0, 1])]

import tensorflow as tf
x = tf.ragged.constant([[0], [0, 1]])
print(x)  # <tf.RaggedTensor [[0], [0, 1]]>

import torch
# x = torch.Tensor([[0], [0, 1]])  # ValueError
```

Response: PyTorch is implementing something called NestedTensors which seems to have pretty much the same purpose as RaggedTensors in Tensorflow. You can follow the RFC and progress here.


Question:
Accuracy score in pyTorch LSTM I have been running this LSTM tutorial on the wikigold.conll NER data set training_data contains a list of tuples of sequences and tags, for example:
```
training_data = [
    ("They also have a song called \" wake up \"".split(), ["O", "O", "O", "O", "O", "O", "I-MISC", "I-MISC", "I-MISC", "I-MISC"]),
    ("Major General John C. Scheidt Jr.".split(), ["O", "O", "I-PER", "I-PER", "I-PER"])
]
```
And I wrote down this function
```
def predict(indices):
    """Gets a list of indices of training_data, and returns a list of predicted lists of tags"""
    for index in indicies:
        inputs = prepare_sequence(training_data[index][0], word_to_ix)
        tag_scores = model(inputs)
        values, target = torch.max(tag_scores, 1)
        yield target
```
This way I can get the predicted labels for specific indices in the training data. However, how do I evaluate the accuracy score across all training data. Accuracy being, the amount of words correctly classified across all sentences divided by the word count. This is what I came up with, which is extremely slow and ugly:
```
y_pred = list(predict([s for s, t in training_data]))
y_true = [t for s, t in training_data]
c=0
s=0
for i in range(len(training_data)):
    n = len(y_true[i])
    #super ugly and ineffiicient
    s+=(sum(sum(list(y_true[i].view(-1, n) == y_pred[i].view(-1, n).data))))
    c+=n

print ('Training accuracy:{a}'.format(a=float(s)/c))
```
How can this be done efficiently in pytorch ? P.S: I've been trying to use sklearn's accuracy_score unsuccessfully

Response:
I would use numpy in order to not iterate the list in pure python. The results are the same, but it runs much faster
```
def accuracy_score(y_true, y_pred):
    y_pred = np.concatenate(tuple(y_pred))
    y_true = np.concatenate(tuple([[t for t in y] for y in y_true])).reshape(y_pred.shape)
    return (y_true == y_pred).sum() / float(len(y_true))
```
And this is how to use it:
```
#original code:
y_pred = list(predict([s for s, t in training_data]))
y_true = [t for s, t in training_data]
#numpy accuracy score
print(accuracy_score(y_true, y_pred))
```


Question:
AttributeError: 'Tensor' has no attribute: 'backwards' I'm posting this question for those who made the same error I did. I got this error when trying to compute my gradients:
```
criterion = torch.nn.CrossEntropyLoss()
loss = criterion(y_hat, y_truth)
loss.backwards()
```

Response: It's loss.backward(), not loss.backwards().


Question:
How do you invert a tensor of boolean values in Pytorch? With NumPy, you can do it with np.invert(array), but there's no invert function in Pytorch. Let's say I have a 2D tensor of boolean values:
```
import torch

ts = torch.rand((10, 4)) < .5
```
```
tensor([[ True,  True, False,  True],
        [ True,  True,  True,  True],
        [ True, False,  True,  True],
        [False,  True,  True, False],
        [False,  True,  True,  True],
        [ True,  True,  True,  True],
        [ True, False,  True,  True],
        [False,  True, False,  True],
        [ True,  True, False,  True],
        [False, False,  True, False]])
```
How do I transform the False into True and vice versa?

Response:
Literally just use the tilde to transform all True into False and vice versa.
```
ts = ~ts
```


Question:
What output_padding does in nn.ConvTranspose2d? What is the working of Output_padding in Conv2dTranspose? Please Help me to understand this?
```
Conv2dTranspose(1024, 512, kernel_size=3, stride=2, padding=1, output_padding=1)
```

Response:
According to documentation here: https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose2d.html when applying Conv2D operation with Stride > 1 you can get same output dimensions with different inputs. For example, 7x7 and 8x8 inputs would both return 3x3 output with Stride=2:
```
import torch

conv_inp1 = torch.rand(1,1,7,7)
conv_inp2 = torch.rand(1,1,8,8)

conv1 = torch.nn.Conv2d(1, 1, kernel_size = 3, stride = 2)

out1 = conv1(conv_inp1)
out2 = conv1(conv_inp2)
print(out1.shape)         # torch.Size([1, 1, 3, 3])
print(out2.shape)         # torch.Size([1, 1, 3, 3])
```
And when applying the transpose convolution, it is ambiguous that which output shape to return, 7x7 or 8x8 for stride=2 transpose convolution. Output padding helps pytorch to determine 7x7 or 8x8 output with output_padding parameter. Note that, it doesn't pad zeros or anything to output, it is just a way to determine the output shape and apply transpose convolution accordingly.
```
conv_t1 = torch.nn.ConvTranspose2d(1, 1, kernel_size=3, stride=2)
conv_t2 = torch.nn.ConvTranspose2d(1, 1, kernel_size=3, stride=2, output_padding=1)
transposed1 = conv_t1(out1)
transposed2 = conv_t2(out2)

print(transposed1.shape)      # torch.Size([1, 1, 7, 7])
print(transposed2.shape)      # torch.Size([1, 1, 8, 8])
```


Question: PyTorch code stops with message "Killed". What killed it? I train a network on GPU with Pytorch. However, after at most 3 epochs, code stops with a message : Killed No other error message is given. I monitored the memory and gpu usage, there were still space during the run. I reviewed the /var/sys/dmesg to find a detailed message regarding to this, however no message with "kill" was entered. What might be the problem? Cuda version: 9.0 Pytorch version: 1.1.0

Response: If you had root access, you could check whether this is memory issue or not by dmesg command. In my case, the process was killed by kernel due to out of memory. I found the cause to be saving tensors require grad to a list and each of those stores an entire computation graph, which consumes significant memory. I fixed the issue by saving .detach() tensor instead of saving tensors returned by loss function to the list.


Question: BatchNorm momentum convention PyTorch Is the batchnorm momentum convention (default=0.1) correct as in other libraries e.g. Tensorflow it seems to usually be 0.9 or 0.99 by default? Or maybe we are just using a different convention?

Response:
It seems that the parametrization convention is different in pytorch than in tensorflow, so that 0.1 in pytorch is equivalent to 0.9 in tensorflow. To be more precise: In Tensorflow:
```
running_mean = decay*running_mean + (1-decay)*new_value
```
In PyTorch:
```
running_mean = (1-decay)*running_mean + decay*new_value
```
This means that a value of decay in PyTorch is equivalent to a value of (1-decay) in Tensorflow.


Question:
Meaning of grad_outputs in PyTorch's torch.autograd.grad I am having trouble understanding the conceptual meaning of the grad_outputs option in torch.autograd.grad. The documentation says: grad_outputs should be a sequence of length matching output containing the “vector” in Jacobian-vector product, usually the pre-computed gradients w.r.t. each of the outputs. If an output doesn’t require_grad, then the gradient can be None). I find this description quite cryptic. What exactly do they mean by Jacobian-vector product? I know what the Jacobian is, but not sure about what product they mean here: element-wise, matrix product, something else? I can't tell from my example below. And why is "vector" in quotes? Indeed, in the example below I get an error when grad_outputs is a vector, but not when it is a matrix.
```

```python
x = torch.tensor([1.,2.,3.,4.], requires_grad=True)
```

```python
y = torch.outer(x, x)
#Output
#```
#Why do we observe the following output; how was it computed?
#```
```

```python
y
#Output
#tensor([[ 1.,  2.,  3.,  4.],
#        [ 2.,  4.,  6.,  8.],
#        [ 3.,  6.,  9., 12.],
#        [ 4.,  8., 12., 16.]], grad_fn=<MulBackward0>)
```

```python
torch.autograd.grad(y, x, grad_outputs=torch.ones_like(y))
#Output
#(tensor([20., 20., 20., 20.]),)
#```
#However, why this error?
#```
```

```python
torch.autograd.grad(y, x, grad_outputs=torch.ones_like(x))
#Output
#```
#RuntimeError: Mismatch in shape: grad_output[0] has a shape of torch.Size([4]) and output[0] has a shape of torch.Size([4, 4]).
```

Response:
If we take your example we have function f which takes as input x shaped (n,) and outputs y = f(x) shaped (n, n). The input is described as column vector [x_i]_i for i ∈ [1, n], and f(x) is defined as matrix [y_jk]_jk = [x_j*x_k]_jk for j, k ∈ [1, n]². It is often useful to compute the gradient of the output with respect to the input (or sometimes w.r.t the parameters of f, there are none here). In the more general case though, we are looking to compute dL/dx and not just dy/dx, where dL/dx is the partial derivative of L, computed from y, w.r.t. x. The computation graph looks like:
```
x.grad = dL/dx <-------   dL/dy y.grad
                dy/dx
       x       ------->    y = x*xT
```
Then, if we look at dL/dx, which is, via the chain rule equal to dL/dy*dy/dx. We have, looking at the interface of torch.autograd.grad, the following correspondences: outputs  y, inputs  x, and grad_outputs  dL/dy. Looking at the shapes: dL/dx should have the same shape as x (dL/dx can be referred to as the 'gradient' of x), while dy/dx, the Jacobian matrix, would be 3-dimensional. On the other hand dL/dy, which is the incoming gradient, should have the same shape as the output, i.e., y's shape. We want to compute dL/dx = dL/dy*dy/dx. If we look more closely, we have
```
dy/dx = [dy_jk/dx_i]_ijk for i, j, k ∈ [1, n]³
```
Therefore,
```
dL/dx = [dL/d_x_i]_i, i ∈ [1,n]
      = [sum(dL/dy_jk * d(y_jk)/dx_i over j, k ∈ [1, n]²]_i, i ∈ [1,n]
```
Back to your example, it means for a given i ∈ [1, n]: dL/dx_i = sum(dy_jk/dx_i) over j, k ∈ [1,n]². And dy_jk/dx_i = f(x_j*x_k)/dx_i will equal x_j if i = k, x_k if i = j, and 2*x_i if i = j = k (because of the squared x_i). This being said matrix y is symmetric... So the result comes down to 2*sum(x_i) over i ∈ [1, n] This means dL/dx is the column vector [2*sum(x)]_i for i ∈ [1, n].
```

```python
2*x.sum()*torch.ones_like(x)
#Output
#tensor([20., 20., 20., 20.])
#```
#Stepping back look at this other graph example, here adding an additional operation after y:
#```
#x   ------->  y = x*xT  -------->  z = y²
#```
#If you look at the backward pass on this graph, you have:
#```
#dL/dx <-------   dL/dy    <--------  dL/dz
#        dy/dx              dz/dy
#  x   ------->  y = x*xT  -------->  z = y²
#```
#With dL/dx = dL/dy*dy/dx = dL/dz*dz/dy*dy/dx which is in practice computed in two sequential steps: dL/dy = dL/dz*dz/dy, then dL/dx = dL/dy*dy/dx.
```


Question:
RuntimeError: Given groups=1, weight of size [32, 3, 16, 16, 16], expected input[100, 16, 16, 16, 3] to have 3 channels, but got 16 channels instead RuntimeError: Given groups=1, weight of size [32, 3, 16, 16, 16], expected input[100, 16, 16, 16, 3] to have 3 channels, but got 16 channels instead This is the portion of code I think where the problem is.
```
def __init__(self):
        super(Lightning_CNNModel, self).__init__()

        self.conv_layer1 = self._conv_layer_set(3, 32)
        self.conv_layer2 = self._conv_layer_set(32, 64)
        self.fc1 = nn.Linear(2**3*64, 128)
        self.fc2 = nn.Linear(128, 10)   # num_classes = 10
        self.relu = nn.LeakyReLU()
        self.batch=nn.BatchNorm1d(128)
        self.drop=nn.Dropout(p=0.15)

    def _conv_layer_set(self, in_c, out_c):
        conv_layer = nn.Sequential(
            nn.Conv3d(in_c, out_c, kernel_size=(3, 3, 3), padding=0),
            nn.LeakyReLU(),
            nn.MaxPool3d((2, 2, 2)),
        )

        return conv_layer

    def forward(self, x):
        out = self.conv_layer1(x)
        out = self.conv_layer2(out)
        out = out.view(out.size(0), -1)
        out = self.fc1(out)
        out = self.relu(out)
        out = self.batch(out)
        out = self.drop(out)
        out = self.fc2(out)

        return out
```
This is the code I am working on

Response:
nn.Conv3d expects the input to have size [batch_size, channels, depth, height, width]. The first convolution expects 3 channels, but with your input having size [100, 16, 16, 16, 3], that would be 16 channels. Assuming that your data is given as [batch_size, depth, height, width, channels], you need to swap the dimensions around, which can be done with torch.Tensor.permute:
```py
# From: [batch_size, depth, height, width, channels]
# To: [batch_size, channels, depth, height, width]
input = input.permute(0, 4, 1, 2, 3)
```


Question: torch.nn.functional vs torch.nn - Pytorch While adding loss in Pytorch, I have the same function in torch.nn.Functional as well as in torch.nn. what is the difference ? torch.nn.CrossEntropyLoss() and torch.nn.functional.cross_entropy

Response: Putting same text from PyTorch discussion forum @Alban D has given answer to similar question. F.cross entropy vs torch.nn.Cross_Entropy_Loss There isn’t much difference for losses. The main difference between the nn.functional.xxx and the nn.Xxx is that one has a state and one does not. This means that for a linear layer for example, if you use the functional version, you will need to handle the weights yourself (including passing them to the optimizer or moving them to the gpu) while the nn.Xxx version will do all of that for you with .parameters() or .to(device). For loss functions, as no parameters are needed (in general), you won’t find much difference. Except for example, if you use cross entropy with some weighting between your classes, using the nn.CrossEntropyLoss() module, you will give your weights only once while creating the module and then use it. If you were using the functional version, you will need to pass the weights every single time you will use it.


Question:
How do they know mean and std, the input value of transforms.Normalize The question is about the data loading tutorial from the PyTorch website. I don't know how they write the value of mean_pix and std_pix of the in transforms.Normalize without calculation I'm unable to find any explanation relevant to this question on StackOverflow.
```py
import torch
from torchvision import transforms, datasets

data_transform = transforms.Compose([
        transforms.RandomSizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225])
    ])
hymenoptera_dataset = datasets.ImageFolder(root='hymenoptera_data/train',
                                           transform=data_transform)
dataset_loader = torch.utils.data.DataLoader(hymenoptera_dataset,
                                             batch_size=4, shuffle=True,
                                             num_workers=4)
```
The value mean=[0.485,0.456, 0.406] and std=[0.229, 0.224, 0.225] is not obvious to me. How do they get them? And why are they equal to these?

Response: For normalization input[channel] = (input[channel] - mean[channel]) / std[channel], the mean and standard deviation values are to be taken from the training dataset. Here, mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] are the mean and std of Imagenet dataset. On Imagenet, we’ve done a pass on the dataset and calculated per-channel mean/std. check here The pre-trained models available in torchvision for transfer learning were pretrained on Imagenet, so using its mean and std deviation would be fine for fine-tuning your model. If you're trying to train your model from scratch, it would be better to use the mean and std deviation of your training dataset (face dataset in this case). Other than that, in most of the cases, the mean and std of Imagenet suffice for your problem.


Question:
How padded sequences given as packed sequences are dealt by RNN in pytorch? In pytorch, we can give a packed sequence as an input to the RNN. From official doc, input of an RNN can be as follows. input (seq_len, batch, input_size): tensor containing the features of the input sequence. The input can also be a packed variable length sequence. Example
```
packed = torch.nn.utils.rnn.pack_padded_sequence(embedded, input_lengths)
outputs, hidden = self.rnn(packed, hidden)
outputs, output_lengths = torch.nn.utils.rnn.pad_packed_sequence(outputs)
```
Here, embedded is the embedded representation of a batch input. My question is, how the computation is carried out for packed sequences in RNN? How the hidden states are computed for padded sequences in a batch through packed representation?

Response:
For the second question: hidden states at padded sequences will not be computed. To answer how is that happening, let's first see what pack_padded_sequence does for us:
```
from torch.nn.utils.rnn import pad_sequence, pad_packed_sequence, pack_padded_sequence

raw = [ torch.ones(25, 300) / 2,
        torch.ones(22, 300) / 2.3,
        torch.ones(15, 300) / 3.2 ]
padded = pad_sequence(raw)  # size: [25, 3, 300]

lengths = torch.as_tensor([25, 22, 15], dtype=torch.int64)
packed = pack_padded_sequence(padded, lengths)
```
so far we randomly created a three tensor with different length (timestep in the context of RNN) , and we first pad them to the same length, then packed it. Now if we run
```
print(padded.size())
print(packed.data.size()) # packed.data refers to the "packed" tensor
```
we will see:
```
torch.Size([25, 3, 300])
torch.Size([62, 300])
```
Obviously 62 does not come from 25 * 3. So what pack_padded_sequence does is only keep the meaningful timestep of each batch entry according to the lengths tensor we passed to pack_padded_sequence (i.e. if we passed [25, 25, 25] to it, the size of packed.data would still be [75, 300] even though the raw tensor does not change). In short, rnn would no even see the pad timestep with pack_padded_sequence And now let's see what's the difference after we pass padded and packed to rnn
```
rnn = torch.nn.RNN(input_size=300, hidden_size=2)
padded_outp, padded_hn = rnn(padded) # size: [25, 3, 2] / [1, 3, 2]
packed_outp, packed_hn = rnn(packed) # 'PackedSequence' Obj / [1, 3, 2]
undo_packed_outp, _ = pad_packed_sequence(packed_outp)

# return "h_n"
print(padded_hn) # tensor([[[-0.2329, -0.6179], [-0.1158, -0.5430],[ 0.0998, -0.3768]]])
print(packed_hn) # tensor([[[-0.2329, -0.6179], [ 0.5622,  0.1288], [ 0.5683,  0.1327]]]

# the output of last timestep (the 25-th timestep)
print(padded_outp[-1]) # tensor([[[-0.2329, -0.6179], [-0.1158, -0.5430],[ 0.0998, -0.3768]]])
print(undo_packed_outp.data[-1]) # tensor([[-0.2329, -0.6179], [ 0.0000,  0.0000], [ 0.0000,  0.0000]]
```
The values of padded_hn and packed_hn are different since rnn DOES compute the pad for padded yet not for the 'packed' (PackedSequence object), which also can be observed from the last hidden state: all three batch entry in padded got non-zero last hidden state even if its length is less than 25. But for packed, the last hidden state for shorter data is not computed (i.e. 0) p.s. another observation:
```
print([(undo_packed_outp[:, i, :].sum(-1) != 0).sum() for i in range(3)])
```
would give us [tensor(25), tensor(22), tensor(15)], which align to the actual length of our input.


Question:
How does torchvision.transforms.Normalize operate? I don't understand how the normalization in Pytorch works. I want to set the mean to 0 and the standard deviation to 1 across all columns in a tensor x of shape (2, 2, 3). A simple example:
```

```python
x = torch.tensor([[[ 1.,  2.,  3.],
#Output
#                       [ 4.,  5.,  6.]],
```

[[ 7.,  8.,  9.],
                        [10., 11., 12.]]])

```python
norm = transforms.Normalize((0, 0), (1, 1))
```

```python
norm(x)
#Output
#tensor([[[ 1.,  2.,  3.],
#         [ 4.,  5.,  6.]],
```

[[ 7.,  8.,  9.],
         [10., 11., 12.]]])
```
So nothing has changed when applying the normalization transform. Why is that?

Response:
To give an answer to your question, you've now realized that torchvision.transforms.Normalize doesn't work as you had anticipated. That's because it's not meant to: normalize: (making your data range in [0, 1]) nor standardize: making your data's mean=0 and std=1 (which is what you're looking for. The operation performed by T.Normalize is merely a shift-scale transform:
```
output[channel] = (input[channel] - mean[channel]) / std[channel]
```
The parameters names mean and std which seems rather misleading knowing that it is not meant to refer to the desired output statistics but instead any arbitrary values. That's right, if you input mean=0 and std=1, it will give you output = (input - 0) / 1 = input. Hence the result you received where function norm had no effect on your tensor values when you were expecting to get a tensor of mean and variance 0 and 1, respectively. However, providing the correct mean and std parameters, i.e. when mean=mean(data) and std=std(data), then you end up calculating the z-score of your data channel by channel, which is what is usually called 'standardization'. So in order to actually get mean=0 and std=1, you first need to compute the mean and standard deviation of your data. If you do:
```

```python
mean, std = x.mean(), x.std()
#Output
#(tensor(6.5000), tensor(3.6056))
#```
#It will give you the global average, and global standard deviation respectively. Instead, what you want is to measure the 1st and 2nd order statistics per-channel. Therefore, we need to apply torch.mean and torch.std on all dimensions expect dim=1. Both of those functions can receive a tuple of dimensions:
#```
```

```python
mean, std = x.mean((0,2)), x.std((0,2))
#Output
#(tensor([5., 8.]), tensor([3.4059, 3.4059]))
#```
#The above is the correct mean and standard deviation of x measured along each channel. From there you can go ahead and use T.Normalize(mean, std) to correctly transform your data x with the correct shift-scale parameters.
#```
```

```python
norm(x)
#Output
#tensor([[[-1.5254, -1.2481, -0.9707],
#         [-0.6934, -0.4160, -0.1387]],
```

[[ 0.1387,  0.4160,  0.6934],
         [ 0.9707,  1.2481,  1.5254]]])
```


Question:
RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' I'm running into an issue while calculating the loss for my Neural Net. I'm not sure why the program expects a long object because all my Tensors are in float form. I looked at threads with similar errors and the solution was to cast Tensors as floats instead of longs, but that wouldn't work in my case because all my data is already in float form when passed to the network. Here's my code:
```
# Dataloader
from torch.utils.data import Dataset, DataLoader

class LoadInfo(Dataset):
    def __init__(self, prediction, indicator):
        self.prediction = prediction
        self.indicator = indicator
    def __len__(self):
        return len(self.prediction)
    def __getitem__(self, idx):
        data = torch.tensor(self.indicator.iloc[idx, :],dtype=torch.float)
        data = torch.unsqueeze(data, 0)
        label = torch.tensor(self.prediction.iloc[idx, :],dtype=torch.float)
        sample = {'data': data, 'label': label}
        return sample

# Trainloader
test_train = LoadInfo(train_label, train_indicators)
trainloader = DataLoader(test_train, batch_size=64,shuffle=True, num_workers=1,pin_memory=True)

# The Network
class NetDense2(nn.Module):

    def __init__(self):
        super(NetDense2, self).__init__()
        self.rnn1 = nn.RNN(11, 100, 3)
        self.rnn2 = nn.RNN(100, 500, 3)
        self.fc1 = nn.Linear(500, 100)
        self.fc2 = nn.Linear(100, 20)
        self.fc3 = nn.Linear(20, 3)

    def forward(self, x):
        x1, h1 = self.rnn1(x)
        x2, h2 = self.rnn2(x1)
        x = F.relu(self.fc1(x2))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

# Allocate / Transfer to GPU
dense2 = NetDense2()
dense2.cuda()

# Optimizer
import torch.optim as optim
criterion = nn.CrossEntropyLoss()                                 # specify the loss function
optimizer = optim.SGD(dense2.parameters(), lr=0.001, momentum=0.9,weight_decay=0.001)

# Training
dense2.train()
loss_memory = []
for epoch in range(50):  # loop over the dataset multiple times
    running_loss = 0.0
    for i, samp in enumerate(trainloader):
        # get the inputs
        ins = samp['data']
        targets = samp['label']
        tmp = []
        tmp = torch.squeeze(targets.float())
        ins, targets = ins.cuda(),  tmp.cuda()
        # zero the parameter gradients
        optimizer.zero_grad()
        # forward + backward + optimize
        outputs = dense2(ins)
        loss = criterion(outputs, targets)     # The loss
        loss.backward()
        optimizer.step()
        # keep track of loss
        running_loss += loss.data.item()
```
I get the error from above in the line " loss = criterion(outputs, targets) "

Response:
As per the documentation and official example at pytorch webpage, The targets passed to nn.CrossEntropyLoss() should be in torch.long format
```
# official example
import torch
import torch.nn as nn
loss = nn.CrossEntropyLoss()
input = torch.randn(3, 5, requires_grad=True)
target = torch.empty(3, dtype=torch.long).random_(5)

# if you will replace the dtype=torch.float, you will get error

output = loss(input, target)
output.backward()
```
update this line in your code as
```
label = torch.tensor(self.prediction.iloc[idx, :],dtype=torch.long) #updated torch.float to torch.long
```


Question:
How to use CUDA stream in Pytorch? I wanna use CUDA stream in Pytorch to parallel some computations, but I don't know how to do it. For instance, if there's 2 tasks, A and B, need to be parallelized, I wanna do the following things:
```
stream0 = torch.get_stream()
stream1 = torch.get_stream()
with torch.now_stream(stream0):
    // task A
with torch.now_stream(stream1):
    // task B
torch.synchronize()
// get A and B's answer
```
How can I achieve the goal in real python code?

Response:
```
s1 = torch.cuda.Stream()
s2 = torch.cuda.Stream()
# Initialise cuda tensors here. E.g.:
A = torch.rand(1000, 1000, device = 'cuda')
B = torch.rand(1000, 1000, device = 'cuda')
# Wait for the above tensors to initialise.
torch.cuda.synchronize()
with torch.cuda.stream(s1):
    C = torch.mm(A, A)
with torch.cuda.stream(s2):
    D = torch.mm(B, B)
# Wait for C and D to be computed.
torch.cuda.synchronize()
# Do stuff with C and D.
```


Question:
How can torch multiply two 10000*10000 matrices in almost zero time? Why does the speed change so much from 349 ms down to 999 µs? Here is an excerpt from Jupyter:

```python
```py
import torch, numpy as np, datetime
cuda = torch.device('cuda')
```
```

```python
```py
ac = torch.randn(10000, 10000).to(cuda)
bc = torch.randn(10000, 10000).to(cuda)
%time cc = torch.matmul(ac, bc)
print(cc[0, 0], torch.sum(ac[0, :] * bc[:, 0]))
```
Wall time: 349 ms tensor(17.0374, device='cuda:0') tensor(17.0376, device='cuda:0') The time is low but still reasonable (0.35 sec for 1e12 multiplications) But if we repeat the same:
```py
ac = torch.randn(10000, 10000).to(cuda)
bc = torch.randn(10000, 10000).to(cuda)
%time cc = torch.matmul(ac, bc)
print(cc[0, 0], torch.sum(ac[0, :] * bc[:, 0]))
```
Wall time: 999 µs tensor(-78.7172, device='cuda:0') tensor(-78.7173, device='cuda:0') 1e12 multiplications in 1ms?! Why did the time change from 349ms to 1ms? Info: Tested on GeForce RTX 2070; Can be reproduced on Google Colab.
```

Response:
There is already a discussion about this on Discuss PyTorch: Measuring GPU tensor operation speed. I'd like to highlight two comments from that thread: From @apaszke: [...] the GPU executes all operations asynchronously, so you need to insert proper barriers for your benchmarks to be correct From @ngimel: I believe cublas handles are allocated lazily now, which means that first operation requiring cublas will have an overhead of creating cublas handle, and that includes some internal allocations. So there’s no way to avoid it other than calling some function requiring cublas before the timing loop. Basically, you have to synchronize() to have a proper measurement:
```py
import torch

x = torch.randn(10000, 10000).to("cuda")
w = torch.randn(10000, 10000).to("cuda")
# ensure that context initialization finish before you start measuring time
torch.cuda.synchronize()

%time y = x.mm(w.t()); torch.cuda.synchronize()
```
CPU times: user 288 ms, sys: 191 ms, total: 479 ms Wall time: 492 ms
```py
x = torch.randn(10000, 10000).to("cuda")
w = torch.randn(10000, 10000).to("cuda")
# ensure that context initialization finish before you start measuring time
torch.cuda.synchronize()

%time y = x.mm(w.t()); torch.cuda.synchronize()
```
CPU times: user 237 ms, sys: 231 ms, total: 468 ms Wall time: 469 ms


Question:
What exactly does the forward function output in Pytorch? This example is taken verbatim from the PyTorch Documentation. Now I do have some background on Deep Learning in general and know that it should be obvious that the forward call represents a forward pass, passing through different layers and finally reaching the end, with 10 outputs in this case, then you take the output of the forward pass and compute the loss using the loss function one defined. Now, I forgot what exactly the output from the forward() pass yields me in this scenario. I thought that the last layer in a Neural Network should be some sort of activation function like sigmoid() or softmax(), but I did not see these being defined anywhere, furthermore, when I was doing a project now, I found out that softmax() is called later on. So I just want to clarify what exactly is the outputs = net(inputs) giving me, from this link, it seems to me by default the output of a PyTorch model's forward pass is logits?
```
transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        print(outputs)
        break
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')
```

Response:
it seems to me by default the output of a PyTorch model's forward pass is logits As I can see from the forward pass, yes, your function is passing the raw output
```py
def forward(self, x):
  x = self.pool(F.relu(self.conv1(x)))
  x = self.pool(F.relu(self.conv2(x)))
  x = x.view(-1, 16 * 5 * 5)
  x = F.relu(self.fc1(x))
  x = F.relu(self.fc2(x))
  x = self.fc3(x)
  return x
```
So, where is softmax? Right here:
```py
criterion = nn.CrossEntropyLoss()
```
It's a bit masked, but inside this function is handled the softmax computation which, of course, works with the raw output of your last layer This is softmax calculation: where z_i are the raw outputs of the neural network So, in conclusion, there is no activation function in your last input because it's handled by the nn.CrossEntropyLoss class Answering what's the raw output that comes from nn.Linear: The raw output of a neural network layer is the linear combination of the values that come from the neurons of the previous layer


Question:
how to apply gradients manually in pytorch Starting to learn pytorch and was trying to do something very simple, trying to move a randomly initialized vector of size 5 to a target vector of value [1,2,3,4,5]. But my distance is not decreasing!! And my vector x just goes crazy. No idea what I am missing.
```python
import torch
import numpy as np
from torch.autograd import Variable

# regress a vector to the goal vector [1,2,3,4,5]

dtype = torch.cuda.FloatTensor # Uncomment this to run on GPU

x = Variable(torch.rand(5).type(dtype), requires_grad=True)
target = Variable(torch.FloatTensor([1,2,3,4,5]).type(dtype),
requires_grad=False)
distance = torch.mean(torch.pow((x - target), 2))

for i in range(100):
  distance.backward(retain_graph=True)
  x_grad = x.grad
  x.data.sub_(x_grad.data * 0.01)
```

Response:
There are two errors in your code that prevents you from getting the desired results. The first error is that you should put the distance calculation in the loop. Because the distance is the loss in this case. So we have to monitor its change in each iteration. The second error is that you should manually zero out the x.grad because pytorch won't zero out the grad in variable by default. The following is an example code which works as expected:
```python
import torch
import numpy as np
from torch.autograd import Variable
import matplotlib.pyplot as plt

# regress a vector to the goal vector [1,2,3,4,5]

dtype = torch.cuda.FloatTensor # Uncomment this to run on GPU

x = Variable(torch.rand(5).type(dtype), requires_grad=True)
target = Variable(torch.FloatTensor([1,2,3,4,5]).type(dtype),
requires_grad=False)

lr = 0.01 # the learning rate

d = []
for i in range(1000):
  distance = torch.mean(torch.pow((x - target), 2))
  d.append(distance.data)
  distance.backward(retain_graph=True)

  x.data.sub_(lr * x.grad.data)
  x.grad.data.zero_()

print(x.data)

fig, ax = plt.subplots()
ax.plot(d)
ax.set_xlabel("iteration")
ax.set_ylabel("distance")
plt.show()
```
The following is the graph of distance w.r.t iteration We can see that the model converges at about 600 iterations. If we set the learning rate to be higher (e.g, lr=0.1), the model will converge much faster (it takes about 60 iterations, see image below) Now, x becomes something like the following 0.9878 1.9749 2.9624 3.9429 4.9292 which is pretty close to your target of [1, 2, 3, 4, 5].


Question:
How retain_grad() in pytorch works? I found its position changes the grad result in a simple test in pytorch, I want to see grad in a non-leaf tensor, so I use retain_grad():
```
import torch
a = torch.tensor([1.], requires_grad=True)
y = torch.zeros((10))
gt = torch.zeros((10))

y[0] = a
y[1] = y[0] * 2
y.retain_grad()

loss = torch.sum((y-gt) ** 2)
loss.backward()
print(y.grad)
```
it gives me a normal output:
```
tensor([2., 4., 0., 0., 0., 0., 0., 0., 0., 0.])
```
but when I use retain grad() before y[1] and after y[0] is assigned:
```
import torch
a = torch.tensor([1.], requires_grad=True)
y = torch.zeros((10))
gt = torch.zeros((10))

y[0] = a
y.retain_grad()
y[1] = y[0] * 2

loss = torch.sum((y-gt) ** 2)
loss.backward()
print(y.grad)
```
now the output changes to:
```
tensor([10.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])
```
I can't understand the result at all.

Response:
Okay so what's going on is really weird. What .retain_grad() essentially does is convert any non-leaf tensor into a leaf tensor, such that it contains a .grad attribute (since by default, pytorch computes gradients to leaf tensors only). Hence, in your first example, after calling y.retain_grad(), it basically converted y into a leaf tensor with an accessible .grad attribute. However, in your second example, you initially converted the entire y tensor into a leaf tensor; then, you created a non-leaf tensor (y[1]) within your leaf tensor (y), which is what caused the confusion.
```
y = torch.zeros((10))  # y is a non-leaf tensor

y[0] = a  # y[0] is a non-leaf tensor
y.retain_grad()  # y is a leaf tensor (including y[1])
y[1] = y[0] * 2  # y[1] is a non-leaf tensor, BUT y[0], y[2], y[3], ..., y[9] are all leaf tensors!
```
The confusing part is: y[1] after calling y.retain_grad() is now a leaf tensor with a .grad attribute. However, y[1] after the computation (y[1] = y[0] * 2) is now not a leaf tensor with a .grad attribute; it is now treated as a new non-leaf variable/tensor. Therefore, when calling loss.backward(), the Chain rule of the loss w.r.t y, and particularly looking at the Chain rule of the loss w.r.t leaf y[1] now looks something like this:


Question: How to handle odd resolutions in Unet architecture PyTorch I'm implementing a U-Net based architecture in PyTorch. At train time, I've patches of size 256x256 which doesn't cause any problem. However at test time, I've full HD images (1920x1080). This is causing a problem during skip connections. Downsampling 1920x1080 3 times gives 240x135. If I downsample one more time, the resolution becomes 120x68 which when upsampled gives 240x136. Now, I cannot concatenate these two feature maps. How can I solve this? PS: I thought this is a fairly common problem, but I didn't get any solution or even mentioning of this problem anywhere on the web. Am I missing something?

Response:
It is a very common problem in segmentation networks where skip-connections are often involved in the decoding process. Networks usually (depending on the actual architecture) require input size that has side lengths as integer multiples of the largest stride (8, 16, 32, etc.). There are two main ways: Resize input to the nearest feasible size. Pad the input to the next larger feasible size. I prefer (2) because (1) can cause small changes in the pixel level for all the pixels, leading to unnecessary blurriness. Note that we usually need to recover the original shape afterward in both methods. My favorite code snippet for this task (symmetric padding for height/width):
```py
import torch
import torch.nn.functional as F

def pad_to(x, stride):
    h, w = x.shape[-2:]

    if h % stride > 0:
        new_h = h + stride - h % stride
    else:
        new_h = h
    if w % stride > 0:
        new_w = w + stride - w % stride
    else:
        new_w = w
    lh, uh = int((new_h-h) / 2), int(new_h-h) - int((new_h-h) / 2)
    lw, uw = int((new_w-w) / 2), int(new_w-w) - int((new_w-w) / 2)
    pads = (lw, uw, lh, uh)

    # zero-padding by default.
    # See others at https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.pad
    out = F.pad(x, pads, "constant", 0)

    return out, pads

def unpad(x, pad):
    if pad[2]+pad[3] > 0:
        x = x[:,:,pad[2]:-pad[3],:]
    if pad[0]+pad[1] > 0:
        x = x[:,:,:,pad[0]:-pad[1]]
    return x
```
A test snippet:
```py
x = torch.zeros(4, 3, 1080, 1920) # Raw data
x_pad, pads = pad_to(x, 16) # Padded data, feed this to your network
x_unpad = unpad(x_pad, pads) # Un-pad the network output to recover the original shape

print('Original: ', x.shape)
print('Padded: ', x_pad.shape)
print('Recovered: ', x_unpad.shape)
```
Output:
```py
Original:  torch.Size([4, 3, 1080, 1920])
Padded:  torch.Size([4, 3, 1088, 1920])
Recovered:  torch.Size([4, 3, 1080, 1920])
```
Reference: https://github.com/seoungwugoh/STM/blob/905f11492a6692dd0d0fa395881a8ec09b211a36/helpers.py#L33


Question: How to do fully connected batch norm in PyTorch? torch.nn has classes BatchNorm1d, BatchNorm2d, BatchNorm3d, but it doesn't have a fully connected BatchNorm class? What is the standard way of doing normal Batch Norm in PyTorch?

Response:
Ok. I figured it out. BatchNorm1d can also handle Rank-2 tensors, thus it is possible to use BatchNorm1d for the normal fully-connected case. So for example:
```
import torch.nn as nn

class Policy(nn.Module):
def __init__(self, num_inputs, action_space, hidden_size1=256, hidden_size2=128):
    super(Policy, self).__init__()
    self.action_space = action_space
    num_outputs = action_space

    self.linear1 = nn.Linear(num_inputs, hidden_size1)
    self.linear2 = nn.Linear(hidden_size1, hidden_size2)
    self.linear3 = nn.Linear(hidden_size2, num_outputs)
    self.bn1 = nn.BatchNorm1d(hidden_size1)
    self.bn2 = nn.BatchNorm1d(hidden_size2)

def forward(self, inputs):
    x = inputs
    x = self.bn1(F.relu(self.linear1(x)))
    x = self.bn2(F.relu(self.linear2(x)))
    out = self.linear3(x)

    return out
```


Question:
How to write torch.device('cuda' if torch.cuda.is_available() else 'cpu') as a full if else statement? I'm a beginner to Pytorch and wanted to type this statement as a whole if else statement:-
```py
torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```
Can somebody help me?

Response:
Here is the code as a whole if-else statement:
```py
torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```
```py
if torch.cuda.is_available():
    torch.device('cuda')
else:
    torch.device('cpu')
```
Since you probably want to store the device for later, you might want something like this instead:
```
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
```
```py
if torch.cuda.is_available():
    device = torch.device('cuda')
else:
    device = torch.device('cpu')
```
Here is a post and discussion about the ternary operator in Python: https://stackoverflow.com/a/2802748/13985765 From that post:
```py
value_when_true if condition else value_when_false
```


Question:
Why doesn't my simple pytorch network work on GPU device? I built a simple network from a tutorial and I got this error: RuntimeError: Expected object of type torch.cuda.FloatTensor but found type torch.FloatTensor for argument #4 'mat1' Any help? Thank you!
```
import torch
import torchvision

device = torch.device("cuda:0")
root = '.data/'

dataset = torchvision.datasets.MNIST(root, transform=torchvision.transforms.ToTensor(), download=True)

dataloader = torch.utils.data.DataLoader(dataset, batch_size=4)

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.out = torch.nn.Linear(28*28, 10)

    def forward(self, x):
        x = x.view(x.size(0), -1)
        x = self.out(x)
        return x

net = Net()
net.to(device)

for i, (inputs, labels) in enumerate(dataloader):
    inputs.to(device)
    out = net(inputs)
```

Response:
TL;DR This is the fix
```
inputs = inputs.to(device)
```
Why?! There is a slight difference between torch.nn.Module.to() and torch.Tensor.to(): while Module.to() is an in-place operator, Tensor.to() is not. Therefore
```
net.to(device)
```
Changes net itself and moves it to device. On the other hand
```
inputs.to(device)
```
does not change inputs, but rather returns a copy of inputs that resides on device. To use that "on device" copy, you need to assign it into a variable, hence
```
inputs = inputs.to(device)
```


Question: Is there a function in google.colab module to close the runtime Sometimes when I give run in google.colab I cant stay infront of the computer to manually disconnect from the server when the run is complete and the connection stays on even when my run is complete occupying the node for no reason. Is there a function in google.colab so that say I can insert the function to close the connection after some epochs? I am looking for something like colab.disconnect() etc...

Response:
Since September 2022, it is now possible. From @GoogleColab twitter: You can now terminate Colab sessions programmatically with the following code:
```
from google.colab import runtime
runtime.unassign()
```
The new unassign() function marks the currently connected runtime for deletion and disconnects any notebook sessions.


Question:
Convolutional NN for text input in PyTorch I am trying to implement a text classification model using a CNN. As far as I know, for text data, we should use 1d Convolutions. I saw an example in pytorch using Conv2d but I want to know how can I apply Conv1d for text? Or, it is actually not possible? Here is my model scenario:
```
Number of in-channels: 1, Number of out-channels: 128
Kernel size : 3 (only want to consider trigrams)
Batch size : 16
```
So, I will provide tensors of shape,  where 28 is the length of a sentence. I want to use Conv1d which will give me 128 feature maps of length 26 (as I am considering trigrams). I am not sure, how to define nn.Conv1d() for this setting. I can use Conv2d but want to know is it possible to achieve the same using Conv1d?

Response:
This example of Conv1d and Pool1d layers into an RNN resolved my issue. So, I need to consider the embedding dimension as the number of in-channels while using nn.Conv1d as follows.
```
m = nn.Conv1d(200, 10, 2) # in-channels = 200, out-channels = 10
input = Variable(torch.randn(10, 200, 5)) # 200 = embedding dim, 5 = seq length
feature_maps = m(input)
print(feature_maps.size()) # feature_maps size = 10,10,4
```


Question: Pytorch Lightning duplicates main script in ddp mode When I launch my main script on the cluster with ddp mode (2 GPU's), Pytorch Lightning duplicates whatever is executed in the main script, e.g. prints or other logic. I need some extended training logic, which I would like to handle myself. E.g. do something (once!) after Trainer.fit(). But with the duplication of the main script, this doesn't work as I intend. I also tried to wrap it in if __name__ == "__main__", but it doesn't change behavior. How could one solve this problem? Or, how can I use some logic around my Trainer object, without the duplicates?

Response: I have since moved on to use the native "ddp" with multiprocessing in PyTorch. As far as I understand, PytorchLightning (PTL) is just running your main script multiple times on multiple GPU's. This is fine if you only want to fit your model in one call of your script. However, a huge drawback in my opinion is the lost flexibility during the training process. The only way of interacting with your experiment is through these (badly documented) callbacks. Honestly, it is much more flexible and convenient to use native multiprocessing in PyTorch. In the end it was so much faster and easier to implement, plus you don't have to search for ages through PTL documentation to achieve simple things. I think PTL is going in a good direction with removing much of the boiler plate, however, in my opinion, the Trainer concept needs some serious rework. It is too closed in my opinion and violates PTL's own concept of "reorganizing PyTorch code, keep native PyTorch code". If you want to use PTL for easy multi GPU training, I personally would strongly suggest to refrain from using it, for me it was a waste of time, better learn native PyTorch multiprocessing.


Question: Extracting the top-k value-indices from a 1-D Tensor Given a 1-D tensor in Torch (torch.Tensor), containing values which can be compared (say floating point), how can we extract the indices of the top-k values in that tensor? Apart from the brute-force method, I am looking for some API call, that Torch/lua provides, which can perform this task efficiently.

Response:
As of pull request #496 Torch now includes a built-in API named torch.topk. Example:
```
> t = torch.Tensor{9, 1, 8, 2, 7, 3, 6, 4, 5}

-- obtain the 3 smallest elements
> res = t:topk(3)
> print(res)
 1
 2
 3
[torch.DoubleTensor of size 3]

-- you can also get the indices in addition
> res, ind = t:topk(3)
> print(ind)
 2
 4
 6
[torch.LongTensor of size 3]

-- alternatively you can obtain the k largest elements as follow
-- (see the API documentation for more details)
> res = t:topk(3, true)
> print(res)
 9
 8
 7
[torch.DoubleTensor of size 3]
```
At the time of writing the CPU implementation follows a sort and narrow approach (there are plans to improve it in the future). That being said an optimized GPU implementation for cutorch is currently being reviewed.


Question:
Can you reverse a PyTorch neural network and activate the inputs from the outputs? Can we activate the outputs of a NN to gain insight into how the neurons are connected to input features? If I take a basic NN example from the PyTorch tutorials. Here is an example of a f(x,y) training example.
```py
import torch

N, D_in, H, D_out = 64, 1000, 100, 10

x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)

loss_fn = torch.nn.MSELoss(reduction='sum')

learning_rate = 1e-4
for t in range(500):
    y_pred = model(x)
    loss = loss_fn(y_pred, y)
    model.zero_grad()
    loss.backward()
    with torch.no_grad():
        for param in model.parameters():
            param -= learning_rate * param.grad
```
After I've finished training the network to predict y from x inputs. Is it possible to reverse the trained NN so that it can now predict x from y inputs? I don't expect y to match the original inputs that trained the y outputs. So I expect to see what features the model activates on to match x and y. If it is possible, then how do I rearrange the Sequential model without breaking all the weights and connections?

Response:
It is possible but only for very special cases. For a feed-forward network (Sequential) each of the layers needs to be reversible; that means the following arguments apply to each layer separately. The transformation associated with one layer is y = activation(W*x + b) where W is the weight matrix and b the bias vector. In order to solve for x we need to perform the following steps: Reverse activation; not all activation functions have an inverse though. For example the ReLU function does not have an inverse on (-inf, 0). If we used tanh on the other hand we can use its inverse which is 0.5 * log((1 + x) / (1 - x)). Solve W*x = inverse_activation(y) - b for x; for a unique solution to exist W must have similar row and column rank and det(W) must be non-zero. We can control the former by choosing a specific network architecture while the latter depends on the training process. So for a neural network to be reversible it must have a very specific architecture: all layers must have the same number of input and output neurons (i.e. square weight matrices) and the activation functions all need to be invertible. Code: Using PyTorch we will have to do the inversion of the network manually, both in terms of solving the system of linear equations as well as finding the inverse activation function. Consider the following example of a 1-layer neural network (since the steps apply to each layer separately extending this to more than 1 layer is trivial):
```
import torch

N = 10  # number of samples
n = 3   # number of neurons per layer

x = torch.randn(N, n)

model = torch.nn.Sequential(
    torch.nn.Linear(n, n), torch.nn.Tanh()
)

y = model(x)

z = y  # use 'z' for the reverse result, start with the model's output 'y'.
for step in list(model.children())[::-1]:
    if isinstance(step, torch.nn.Linear):
        z = z - step.bias[None, ...]
        z = z[..., None]  # 'torch.solve' requires N column vectors (i.e. shape (N, n, 1)).
        z = torch.solve(z, step.weight)[0]
        z = torch.squeeze(z)  # remove the extra dimension that we've added for 'torch.solve'.
    elif isinstance(step, torch.nn.Tanh):
        z = 0.5 * torch.log((1 + z) / (1 - z))

print('Agreement between x and z: ', torch.dist(x, z))
```


Question:
PyTorch Gradient Descent I am trying to manually implement gradient descent in PyTorch as a learning exercise. I have the following to create my synthetic dataset:
```
import torch
torch.manual_seed(0)
N = 100
x = torch.rand(N,1)*5
# Let the following command be the true function
y = 2.3 + 5.1*x
# Get some noisy observations
y_obs = y + 2*torch.randn(N,1)
```
Then I create my predictive function (y_pred) as shown below.
```
w = torch.randn(1, requires_grad=True)
b = torch.randn(1, requires_grad=True)
y_pred = w*x+b
mse = torch.mean((y_pred-y_obs)**2)
```
which uses MSE to infer the weights w,b. I use the block below to update the values according to the gradient.
```
gamma = 1e-2
for i in range(100):
  w = w - gamma *w.grad
  b = b - gamma *b.grad
  mse.backward()
```
However, the loop only works in the first iteration. The second iteration onwards w.grad is set to None. I am fairly sure the reason this happens is because I am setting w as a function of it self (I might be wrong). The question is how do I update the weights properly with the gradient information?

Response:
You should call the backward method before you apply the gradient descent. You need to use the new weight to calculate the loss every iteration. Create new tensor without gradient tape every iteration. The following code works fine on my computer and gives w=5.1 & b=2.2 after 500 iterations training. Code:
```
import torch
torch.manual_seed(0)
N = 100
x = torch.rand(N,1)*5
# Let the following command be the true function
y = 2.3 + 5.1*x
# Get some noisy observations
y_obs = y + 0.2*torch.randn(N,1)

w = torch.randn(1, requires_grad=True)
b = torch.randn(1, requires_grad=True)

gamma = 0.01
for i in range(500):
    print(i)
    # use new weight to calculate loss
    y_pred = w * x + b
    mse = torch.mean((y_pred - y_obs) ** 2)

    # backward
    mse.backward()
    print('w:', w)
    print('b:', b)
    print('w.grad:', w.grad)
    print('b.grad:', b.grad)

    # gradient descent, don't track
    with torch.no_grad():
        w = w - gamma * w.grad
        b = b - gamma * b.grad
    w.requires_grad = True
    b.requires_grad = True
```
Output:
```
499
w: tensor([5.1095], requires_grad=True)
b: tensor([2.2474], requires_grad=True)
w.grad: tensor([0.0179])
b.grad: tensor([-0.0576])
```


Question: Shall we lower case input data for (pre) training a BERT uncased model using huggingface? Shall we lower case input data for (pre) training a BERT uncased model using huggingface? I looked into this response from Thomas Wolf (https://github.com/huggingface/transformers/issues/92#issuecomment-444677920) but not entirely sure if he meant that. What happens if we lowercase the text ?

Response:
Tokenizer will take care of that. A simple example:
```
import torch
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', max_length = 10, padding_side = 'right')

input_ids = torch.tensor(tokenizer.encode('this is a cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)

input_ids = torch.tensor(tokenizer.encode('This is a Cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)
```
Out:
```
tensor([[ 101, 2023, 2003, 1037, 4937,  102,    0,    0,    0,    0]])
tensor([[ 101, 2023, 2003, 1037, 4937,  102,    0,    0,    0,    0]])
```
But in case of cased,
```
tokenizer = BertTokenizer.from_pretrained('bert-base-cased', max_length = 10, padding_side = 'right')

input_ids = torch.tensor(tokenizer.encode('this is a cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)

input_ids = torch.tensor(tokenizer.encode('This is a Cat', add_special_tokens=True, max_length = 10, pad_to_max_length = True)).unsqueeze(0)
print(input_ids)
```
```
tensor([[ 101, 1142, 1110,  170, 5855,  102,    0,    0,    0,    0]])

tensor([[ 101, 1188, 1110,  170, 8572,  102,    0,    0,    0,    0]])
```


Question:
RuntimeError: Expected object of backend CUDA but got backend CPU for argument: ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t()) When the forward function of my neural network (after the training phase is completed) is being executed, I'm experiencing RuntimeError: Expected object of backend CUDA but got backend CPU for argument #4 'mat1'. The error trace indicates the error happens due to the call of output = self.layer1(x) command. I have tried to move all the data of the tensors to my GPU. It seems I miss something to be moved as well. Here is the code I have tried:
```
use_cuda = torch.cuda.is_available()
device = torch.device('cuda:0' if use_cuda else 'cpu')

class NeuralNet(nn.Module):

    def __init__(self, input_size, hidden_size, output_size):
        super(NeuralNet, self).__init__()
        self.layer1 = nn.Linear(input_size, hidden_size).cuda(device)
        self.layer2 = nn.Linear(hidden_size, output_size).cuda(device)
        self.relu = nn.ReLU().cuda(device)

    def forward(self, x):
        x.cuda(device)
        output = self.layer1(x)  # throws the error
        output = self.relu(output)
        output = self.layer2(output)
        return output

def main():
    transform = transforms.Compose([
        transforms.ToTensor()
    ])

    mnist_trainset = datasets.MNIST(root='D:\\MNIST', train=True, download=False, transform=transform)
    mnist_testset = datasets.MNIST(root='D:\\MNIST', train=False, download=False, transform=transform)

    train_loader = DataLoader(dataset=mnist_trainset, batch_size=100, shuffle=True)
    test_loader = DataLoader(dataset=mnist_testset, batch_size=100, shuffle=False)

    input_size = 784
    hidden_size = 500
    output_size = 10
    num_epochs = 5

    learning_rate = 0.001

    model = NeuralNet(input_size, hidden_size, output_size)
    model.cuda(device)

    lossFunction = nn.CrossEntropyLoss()
    lossFunction.cuda(device)
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

    losses_in_epochs = []
    total_step = len(train_loader)
    for epoch in range(num_epochs):
        for i, (images, labels) in enumerate(train_loader):
            images = images.to(device)
            labels = labels.to(device)
            images = images.reshape(-1, 28 * 28)

            out = model(images)
            loss = lossFunction(out, labels)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            if (i + 1) % 100 == 0:
                print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch + 1, num_epochs, i + 1, total_step,
                                                                         loss.item()))

            if (i % 600) == 0:
                losses_in_epochs.append(loss.item())

    with torch.no_grad():
        correct = 0
        total = 0
        for images, labels in test_loader:
            images = images.reshape(-1, 28 * 28)
            out = model(images)
            _, predicted = torch.max(out.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
            print('Accuracy of the network on the 10000 test images: {} %'.format(100 * correct / total))

if __name__ == '__main__':
    main()
```
The software stack:
```
Python 3.7.1
torch 1.0.1 (with Cuda 9.0)
Windows 10 64-bit
```

Response:
The error only happens only at the testing step, when you try calculating the accuracy, this might already give you a hint. The training loop runs without a problem. The error is simply that you don't send the images and labels to the GPU at this step. This is your corrected evaluation loop:
```
with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in test_loader:
        images = images.to(device)  # missing line from original code
        labels = labels.to(device)  # missing line from original code
        images = images.reshape(-1, 28 * 28)
        out = model(images)
        _, predicted = torch.max(out.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
```
BTW you don't need to send all your layers to the GPU separately (at your class __init__()). It's better to just send the whole instantiated model to the gpu at once.


Question:
How exactly the forward and backward hooks work in PyTorch I am trying to understand how exactly code-wise the hooks operate in PyTorch. I have a model and I would like to set a forward and backward hook in my code. I would like to set a hook in my model after a specific layer and I guess the easiest way is to set a hook to this specific module. This introductory video warns that the backward module contains a bug, but I am not sure if that is still the case. My code looks as follows:
```
def __init__(self, model, attention_layer_name='desired_name_module',discard_ratio=0.9):
  self.model = model
  self.discard_ratio = discard_ratio
  for name, module in self.model.named_modules():
    if attention_layer_name in name:
        module.register_forward_hook(self.get_attention)
        module.register_backward_hook(self.get_attention_gradient)

  self.attentions = []
  self.attention_gradients = []

def get_attention(self, module, input, output):
  self.attentions.append(output.cpu())

def get_attention_gradient(self, module, grad_input, grad_output):
  self.attention_gradients.append(grad_input[0].cpu())

def __call__(self, input_tensor, category_index):
  self.model.zero_grad()
  output = self.model(input_tensor)
  loss = ...
  loss.backward()
```
I am puzzled to understand how code-wise the following lines work:
```
module.register_forward_hook(self.get_attention)
module.register_backward_hook(self.get_attention_gradient)
```
I am registering a hook to my desired module, however, then, I am calling a function in each case without any input. My question is Python-wise, how does this call work exactly? How the arguments of the register_forward_hook and register_backward_hook operate when the function it's called?

Response:
How does a hook work? A hook allows you to execute a specific function - referred to as a "callback" - when a particular action has been performed. In this case, you are expecting self.get_attention to be called once the forward function of module has been accessed. To give a minimal example of how a hook would look like. I define a simple class on which you can register new callbacks through register_hook, then when the instance is called (via __call__), all hooks will be called with the provided arguments:
```
class Obj:
    def __init__(self):
        self.hooks = []

    def register_hook(self, hook):
        self.hooks.append(hook)

    def __call__(self, x, y):
        print('instance called')
        for hook in self.hooks:
            hook(x, y)
```
First, implement two hooks for demonstration purposes:
```
def foo(x, y):
    print(f'foo called with {x} and {y}')
def bar(x, _):
    print(f'bar called with {x}')
```
And initialize an instance of Obj:
```
obj = Obj()
```
You can register a hook and call the instance:
```

```python
obj.register_hook(foo)
```

```python
obj('yes', 'no')
#Output
#instance called
#foo called with yes and no
#```
#You can add hooks on top and call again to compare, here both hooks are triggered:
#```
```

```python
obj.register_hook(bar)
```

```python
obj('yes', 'no')
#Output
#instance called
#foo called with yes and no
#bar called with yes
#```
#Using hooks in PyTorch There are two primary hooks in PyTorch: forward and backward. You also have pre- and post-hooks. Additionally there exists hooks on other actions such as load_state_dict... To attach a hook on the forward process of a nn.Module, you should use register_forward_hook, the argument is a callback function that expects module, args, and output. This callback will be triggered on every forward execution. For backward hooks, you should use register_full_backward_hook, the registered hook expects three arguments: module, grad_input, and grad_output. As of recent PyTorch versions, register_backward_hook has been deprecated and should not be used. One side effect here is that you are registering the hook with self.get_attention and self.get_attention_gradient. The function passed to the register handler is not unbound to the class instance! In other words, on execution, these will be called without the self argument like:
#```
#self.get_attention(module, input, output)
#self.get_attention_gradient(module, grad_input, grad_output)
#```
#This will fail. A simple way to fix this is to wrap the hook with a lambda when you register it:
#```
#module.register_forward_hook(
#    lambda *args, **kwargs: Routine.get_attention(self, *args, **kwargs))
#```
#All in all, your class could look like this:
#```
#class Routine:
#    def __init__(self, model, attention_layer_name):
#        self.model = model
```

for name, module in self.model.named_modules():
            if attention_layer_name in name:
                module.register_forward_hook(
                    lambda *args, **kwargs: Routine.get_attention(self, *args, **kwargs))
                module.register_full_backward_hook(
                    lambda *args, **kwargs: Routine.get_attention_gradient(self, *args, **kwargs))

        self.attentions = []
        self.attention_gradients = []

    def get_attention(self, module, input, output):
        self.attentions.append(output.cpu())

    def get_attention_gradient(self, module, grad_input, grad_output):
        self.attention_gradients.append(grad_input[0].cpu())

    def __call__(self, input_tensor):
        self.model.zero_grad()
        output = self.model(input_tensor)
        loss = output.mean()
        loss.backward()
```
When initialized with a single linear layer model:
```
routine = Routine(nn.Sequential(nn.Linear(10,10)), attention_layer_name='0')
```
You can call the instance, this will first trigger the forward hook with (because of self.model(input_tensor), and then the backward hook (because of loss.backward()).
```

```python
routine(torch.rand(1,10, requires_grad=True))
#Output
#```
#Following your implementation, your forward hook is caching the output of the "attention_layer_name" layer in self.attentions.
#```
```

```python
routine.attentions
#Output
#[tensor([[-0.3137, -0.2265, -0.2197,  0.2211, -0.6700,
#          -0.5034, -0.1878, -1.1334,  0.2025,  0.8679]], grad_fn=<...>)]
#```
#Similarly for the self.attention_gradients:
#```
```

```python
routine.attentions_gradients
#Output
#[tensor([[-0.0501,  0.0393,  0.0353, -0.0257,  0.0083,
#           0.0426, -0.0004, -0.0095, -0.0759, -0.0213]])]
#```
#It is important to note that the cached outputs and gradients will remain in self.attentions and self.attentions_gradients and get appended on every execution of Routine.__call__.
```


Question:
"RuntimeError: expected scalar type Double but found Float" in Pytorch CNN training I just begin to learn Pytorch and create my first CNN. The dataset contains 3360 RGB images and I converted them to a [3360, 3, 224, 224] tensor. The data and label are in the dataset(torch.utils.data.TensorDataset). Below is the training code.
```py
def train_net():
        dataset = ld.load()
        data_iter = Data.DataLoader(dataset, batch_size=168, shuffle=True)
        net = model.VGG_19()
        summary(net, (3, 224, 224), device="cpu")
        loss_func = nn.CrossEntropyLoss()
        optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9, dampening=0.1)
        scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=50, gamma=0.1)
        for epoch in range(5):
            print("epoch:", epoch + 1)
            train_loss = 0
            for i, data in enumerate(data_iter, 0):
                x, y = data
                print(x.dtype)
                optimizer.zero_grad()
                out = net(x)
                loss = loss_func(out, y)
                loss.backward()
                optimizer.step()
                train_loss += loss.item()
                if i % 100 == 99:
                    print("loss:", train_loss / 100)
                    train_loss = 0.0
        print("finish train")
```
Then I have this error:
```
Traceback (most recent call last):
              File "D:/python/DeepLearning/VGG/train.py", line 52, in <module>
                train_net()
              File "D:/python/DeepLearning/VGG/train.py", line 29, in train_net
                out = net(x)
              File "D:\python\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
                result = self.forward(*input, **kwargs)
              File "D:\python\DeepLearning\VGG\model.py", line 37, in forward
                out = self.conv3_64(x)
              File "D:\python\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
                result = self.forward(*input, **kwargs)
              File "D:\python\lib\site-packages\torch\nn\modules\container.py", line 117, in forward
                input = module(input)
              File "D:\python\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
                result = self.forward(*input, **kwargs)
              File "D:\python\lib\site-packages\torch\nn\modules\conv.py", line 423, in forward
                return self._conv_forward(input, self.weight)
              File "D:\python\lib\site-packages\torch\nn\modules\conv.py", line 419, in _conv_forward
                return F.conv2d(input, weight, self.bias, self.stride,
            RuntimeError: expected scalar type Double but found Float
```
I think there is something wrong with x and I print its type by print(x.dtype):
```
torch.float64
```
which is double instead of float. Do you know what`s wrong? Thanks for your help!

Response:
that error is actually refering to the weights of the conv layer which are in float32 by default when the matrix multiplication is called. Since your input is double(float64 in pytorch) while the weights in conv are float So the solution in your case is :
```
def train_net():
    dataset = ld.load()
    data_iter = Data.DataLoader(dataset, batch_size=168, shuffle=True)
    net = model.VGG_19()
    summary(net, (3, 224, 224), device="cpu")
    loss_func = nn.CrossEntropyLoss()
    optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9, dampening=0.1)
    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=50, gamma=0.1)
    for epoch in range(5):
        print("epoch:", epoch + 1)
        train_loss = 0
        for i, data in enumerate(data_iter, 0):
            x, y = data                        #      //_______________
            x = x.float()    # HERE IS THE CHANGE     \\
            print(x.dtype)
            optimizer.zero_grad()
            out = net(x)
            loss = loss_func(out, y)
            loss.backward()
            optimizer.step()
            train_loss += loss.item()
            if i % 100 == 99:
                print("loss:", train_loss / 100)
                train_loss = 0.0
    print("finish train")
```
This will work for sure


Question:
Why do we do batch matrix-matrix product? I'm following Pytorch seq2seq tutorial and ittorch.bmm method is used like below:
```
attn_applied = torch.bmm(attn_weights.unsqueeze(0),
                         encoder_outputs.unsqueeze(0))
```
I understand why we need to multiply attention weight and encoder outputs. What I don't quite understand is the reason why we need bmm method here. torch.bmm document says Performs a batch matrix-matrix product of matrices stored in batch1 and batch2. batch1 and batch2 must be 3-D tensors each containing the same number of matrices. If batch1 is a (b×n×m) tensor, batch2 is a (b×m×p) tensor, out will be a (b×n×p) tensor.

Response:
In the seq2seq model, the encoder encodes the input sequences given in as mini-batches. Say for example, the input is B x S x d where B is the batch size, S is the maximum sequence length and d is the word embedding dimension. Then the encoder's output is B x S x h where h is the hidden state size of the encoder (which is an RNN). Now while decoding (during training) the input sequences are given one at a time, so the input is B x 1 x d and the decoder produces a tensor of shape B x 1 x h. Now to compute the context vector, we need to compare this decoder hidden state with the encoder's encoded states. So, consider you have two tensors of shape T1 = B x S x h and T2 = B x 1 x h. So if you can do batch matrix multiplication as follows.
```
out = torch.bmm(T1, T2.transpose(1, 2))
```
Essentially you are multiplying a tensor of shape B x S x h with a tensor of shape B x h x 1 and it will result in B x S x 1 which is the attention weight for each batch. Here, the attention weights B x S x 1 represent a similarity score between the decoder's current hidden state and encoder's all the hidden states. Now you can take the attention weights to multiply with the encoder's hidden state B x S x h by transposing first and it will result in a tensor of shape B x h x 1. And if you perform squeeze at dim=2, you will get a tensor of shape B x h which is your context vector. This context vector (B x h) is usually concatenated to decoder's hidden state (B x 1 x h, squeeze dim=1) to predict the next token.