Question:
How to implement dropout in Pytorch, and where to apply it I am quite unsure whether this is correct. It is really sad I can't find many good examples on how to parametrize a NN. What do you think of this way of dropping out in those two classes. First I'm writing the original class :
```
class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, num_classes, p = dropout):
      super(NeuralNet, self).__init__()
      self.fc1 = nn.Linear(input_size, hidden_size)
      self.fc2 = nn.Linear(hidden_size, hidden_size)
      self.fc3 = nn.Linear(hidden_size, num_classes)

  def forward(self, x):
      out = F.relu(self.fc1(x))
      out = F.relu(self.fc2(out))
      out = self.fc3(out)
      return out
```
and then here, I found two different ways to write things, which I don't know how to distinguish. The first one uses :
```
self.drop_layer = nn.Dropout(p=p)
```
whereas the second :
```
self.dropout = nn.Dropout(p)
```
and here is my result :
```
class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, num_classes, p = dropout):
      super(NeuralNet, self).__init__()
      self.fc1 = nn.Linear(input_size, hidden_size)
      self.fc2 = nn.Linear(hidden_size, hidden_size)
      self.fc3 = nn.Linear(hidden_size, num_classes)
      self.drop_layer = nn.Dropout(p=p)

  def forward(self, x):
      out = F.relu(self.fc1(x))
      out = F.relu(self.fc2(out))
      out = self.fc3(out)
      return out

 class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, num_classes, p = dropout):
      super(NeuralNet, self).__init__()
      self.fc1 = nn.Linear(input_size, hidden_size)
      self.fc2 = nn.Linear(hidden_size, hidden_size)
      self.fc3 = nn.Linear(hidden_size, num_classes)
      self.dropout = nn.Dropout(p)

  def forward(self, x):
      out = F.relu(self.fc1(x))
      out = F.relu(self.fc2(out))
      out = self.fc3(out)
      return out
```
could this work, if not how to improve that, and is it giving me the result I m expecting, meaning creating a neural network where I can dropout some neurons. Important detail, I only want to dropout the second layer of neural network, no touch to the rest!

Response:
The two examples you provided are exactly the same. self.drop_layer = nn.Dropout(p=p) and self.dropout = nn.Dropout(p) only differ because the authors assigned the layers to different variable names. The dropout layer is typically defined in the .__init__() method, and called in .forward(). Like this:
```py
class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, num_classes, p = dropout):
      super(NeuralNet, self).__init__()
      self.fc1 = nn.Linear(input_size, hidden_size)
      self.fc2 = nn.Linear(hidden_size, hidden_size)
      self.fc3 = nn.Linear(hidden_size, num_classes)
      self.dropout = nn.Dropout(p)

  def forward(self, x):
      out = F.relu(self.fc1(x))
      out = F.relu(self.fc2(out))
      out = self.dropout(self.fc3(out))
      return out
```
You can do the test:
```py
import torch
import torch.nn  as nn

m = nn.Dropout(p=0.5)
input = torch.randn(20, 16)
print(torch.sum(torch.nonzero(input)))
print(torch.sum(torch.nonzero(m(input))))
```
```py
tensor(5440) # sum of nonzero values
tensor(2656) # sum on nonzero values after dropout
```
Let's visualize it:
```py
import torch
import torch.nn as nn
input = torch.randn(5, 5)
print(input)
```
```py
tensor([[ 1.1404,  0.2102, -0.1237,  0.4240,  0.0174],
        [-2.0872,  1.2790,  0.7804, -0.0962, -0.9730],
        [ 0.4788, -1.3408,  0.0483,  2.4125, -1.2463],
        [ 1.5761,  0.3592,  0.2302,  1.3980,  0.0154],
        [-0.4308,  0.2484,  0.8584,  0.1689, -1.3607]])
```
Now, let's apply the dropout:
```py
m = nn.Dropout(p=0.5)
output = m(input)
print(output)
```
```py
tensor([[ 0.0000,  0.0000, -0.0000,  0.8481,  0.0000],
        [-0.0000,  0.0000,  1.5608, -0.0000, -1.9459],
        [ 0.0000, -0.0000,  0.0000,  0.0000, -0.0000],
        [ 0.0000,  0.7184,  0.4604,  2.7959,  0.0308],
        [-0.0000,  0.0000,  0.0000,  0.0000, -0.0000]])
```
Approximately half the neurons have been turned to zero, because we had probability p=0.5 that a neuron is set to zero!


Question:
PyTorch get all layers of model What's the easiest way to take a pytorch model and get a list of all the layers without any nn.Sequence groupings? For example, a better way to do this?
```
import pretrainedmodels

def unwrap_model(model):
    for i in children(model):
        if isinstance(i, nn.Sequential): unwrap_model(i)
        else: l.append(i)

model = pretrainedmodels.__dict__['xception'](num_classes=1000, pretrained='imagenet')
l = []
unwrap_model(model)

print(l)
```

Response:
You can iterate over all modules of a model (including those inside each Sequential) with the modules() method. Here's a simple example:
```

```python
model = nn.Sequential(nn.Linear(2, 2),
#Output
#                          nn.ReLU(),
#                          nn.Sequential(nn.Linear(2, 1),
#                          nn.Sigmoid()))
```

```python
l = [module for module in model.modules() if not isinstance(module, nn.Sequential)]
```

```python
l
```

[Linear(in_features=2, out_features=2, bias=True),
 ReLU(),
 Linear(in_features=2, out_features=1, bias=True),
 Sigmoid()]
```


Question:
what does dim=-1 or -2 mean in torch.sum()? let me take a 2D matrix as example:
```
mat = torch.arange(9).view(3, -1)

tensor([[0, 1, 2],
        [3, 4, 5],
        [6, 7, 8]])

torch.sum(mat, dim=-2)

tensor([ 9, 12, 15])
```
I find the result of torch.sum(mat, dim=-2) is equal to torch.sum(mat, dim=0) and dim=-1 equal to dim=1. My question is how to understand the negative dimension here. What if the input matrix has 3 or more dimensions?

Response: A tensor has multiple dimensions, ordered as in the following figure. There is a forward and backward indexing. Forward indexing uses positive integers, backward indexing uses negative integers. Example: -1 will be the last one, in our case it will be dim=2 -2 will be dim=1 -3 will be dim=0


Question: difference between Dataset and TensorDataset in pyTorch what is the difference between "torch.utils.data.TensorDataset" and "torch.utils.data.Dataset" - the docs are not clear about that and I could not find any answers on google.

Response:
The Dataset class is an abstract class that is used to define new types of (customs) datasets. Instead, the TensorDataset is a ready to use class to represent your data as list of tensors. You can define your custom dataset in the following way:
```py
class CustomDataset(torch.utils.data.Dataset):

  def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    # Your code

    self.instances = your_data

  def __getitem__(self, idx):
    return self.instances[idx] # In case you stored your data on a list called instances

  def __len__(self):
    return len(self.instances)
```
If you just want to create a dataset that contains tensors for input features and labels, then use the TensorDataset directly:
```py
dataset = TensorDataset(input_features, labels)
```
Note that input_features and labels must match on the length of the first dimension.


Question: In PyTorch, what exactly does the grad_fn attribute store and how is it used? In PyTorch, the Tensor class has a grad_fn attribute. This references the operation used to obtain the tensor: for instance, if a = b + 2, a.grad_fn will be AddBackward0. But what does "reference" mean exactly? Inspecting AddBackward0 using inspect.getmro(type(a.grad_fn)) will state that the only base class of AddBackward0 is object. Additionally, the source code for this class (and in fact, any other class which might be encountered in grad_fn) is nowhere to be found in the source code! All of this leads me to the following questions: What precisely is stored in grad_fn and how is it called during back-propagation? How come the objects that get stored in grad_fn do not have some sort of common super class, and why is there no source code for them on GitHub?

Response: grad_fn is a function "handle", giving access to the applicable gradient function. The gradient at the given point is a coefficient for adjusting weights during back-propagation. "Handle" is a general term for an object descriptor, designed to give appropriate access to the object. For instance, when you open a file, open returns a file handle. When you instantiate a class, the __init__ function returns a handle to the created instance. The handle contains references (usually memory addresses) to the data and functions for the item in question. It appears as the generic object class because it's from the underlying implementation in another language, such that it does not map exactly to the Python function type. PyTorch handles the inter-language call and return. This hand-off is part of the pre-complied (shared-object) run-time system. Is that enough to clarify what you see?


Question:
How does pytorch broadcasting work?
```
torch.add(torch.ones(4,1), torch.randn(4))
```
produces a Tensor with size: torch.Size([4,4]). Can someone provide a logic behind this?

Response:
PyTorch broadcasting is based on numpy broadcasting semantics which can be understood by reading numpy broadcasting rules or PyTorch broadcasting guide. Expounding the concept with an example would be intuitive to understand it better. So, please see the example below:
```

```python
t_rand
#Output
#tensor([ 0.23451,  0.34562,  0.45673])
```

```python
t_ones
#Output
#tensor([[ 1.],
#        [ 1.],
#        [ 1.],
#        [ 1.]])
#```
#Now for torch.add(t_rand, t_ones), visualize it like:
#```
## shape of (3,)
#               tensor([ 0.23451,      0.34562,       0.45673])
#      # (4, 1)          | | | |       | | | |        | | | |
#      tensor([[ 1.],____+ | | |   ____+ | | |    ____+ | | |
#              [ 1.],______+ | |   ______+ | |    ______+ | |
#              [ 1.],________+ |   ________+ |    ________+ |
#              [ 1.]])_________+   __________+    __________+
#```
#which should give the output with tensor of shape (4,3) as:
#```
## shape of (4,3)
```

```python
torch.add(t_rand, t_ones)
#Output
#tensor([[ 1.23451,  1.34562,  1.45673],
#        [ 1.23451,  1.34562,  1.45673],
#        [ 1.23451,  1.34562,  1.45673],
#        [ 1.23451,  1.34562,  1.45673]])
#```
#Also, note that we get exactly the same result even if we pass the arguments in a reverse order as compared to the previous one:
#```
## shape of (4, 3)
```

```python
torch.add(t_ones, t_rand)
#Output
#tensor([[ 1.23451,  1.34562,  1.45673],
#        [ 1.23451,  1.34562,  1.45673],
#        [ 1.23451,  1.34562,  1.45673],
#        [ 1.23451,  1.34562,  1.45673]])
#```
#Anyway, I prefer the former way of understanding for more straightforward intuitiveness. For pictorial understanding, I culled out more examples which are enumerated below: Example-1: Example-2:: T and F stand for True and False respectively and indicate along which dimensions we allow broadcasting (source: Theano). Example-3: Here are some shapes where the array b is broadcasted appropriately to attempt to match the shape of the array a. As shown above, the broadcasted b may still not match the shape of a, and so the operation a + b will fail whenever the final broadcasted shapes do not match.
```


Question: Why can GPU do matrix multiplication faster than CPU? I've been using GPU for a while without questioning it but now I'm curious. Why can GPU do matrix multiplication much faster than CPU? Is it because of parallel processing? But I didn't write any parallel processing code. Does it do it automatically by itself? Any intuition / high-level explanation will be appreciated!

Response:
How do you parallelize the computations? GPU's are able to do a lot of parallel computations. A Lot more than a CPU could do. Look at this example of vector addition of let's say 1M elements. Using a CPU let's say you have 100 maximum threads you can run : (100 is lot more but let's assume for a while) In a typical multi-threading example let's say you parallelized additions on all threads. Here is what I mean by it :
```
c[0] = a[0] + b[0] # let's do it on thread 0
c[1] = a[1] + b[1] # let's do it on thread 1
c[101] = a[101] + b[101] # let's do it on thread 1
```
We are able to do it because value of c[0], doesn't depend upon any other values except a[0] and b[0]. So each addition is independent of others. Hence, we were able to easily parallelize the task. As you see in above example that simultaneously all the addition of 100 different elements take place saving you time. In this way it takes 1M/100 = 10,000 steps to add all the elements. How Efficient does GPU Parallelizes? Now consider today's GPU with about 2048 threads, all threads can independently do 2048 different operations in constant time. Hence giving a boost up. In your case of matrix multiplication. You can parallelize the computations, Because GPU have much more threads and in each thread you have multiple blocks. So a lot of computations are parallelized, resulting quick computations. But I didn't write any parallel processing for my GTX1080! Does it do it by itself? Almost all the framework for machine learning uses parallelized implementation of all the possible operations. This is achieved by CUDA programming, NVIDIA API to do parallel computations on NVIDIA GPU's. You don't write it explicitly, it's all done at low level, and you do not even get to know. Yes it doesn't mean that a C++ program you wrote will automatically be parallelized, just because you have a GPU. No, you need to write it using CUDA, only then it will be parallelized, but most programming framework have it, So it is not required from your end.


Question:
pytorch RuntimeError: Expected object of scalar type Double but got scalar type Float I am trying to implement a custom dataset for my neural network. But got this error when running the forward function. The code is as follows.
```
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
import numpy as np

class ParamData(Dataset):
    def __init__(self,file_name):
        self.data = torch.Tensor(np.loadtxt(file_name,delimiter = ','))    #first place
    def __len__(self):
        return self.data.size()[0]
    def __getitem__(self,i):
        return self.data[i]

class Net(nn.Module):
    def __init__(self,in_size,out_size,layer_size=200):
        super(Net,self).__init__()
        self.layer = nn.Linear(in_size,layer_size)
        self.out_layer = nn.Linear(layer_size,out_size)

    def forward(self,x):
        x = F.relu(self.layer(x))
        x = self.out_layer(x)
        return x

datafile = 'data1.txt'

net = Net(100,1)
dataset = ParamData(datafile)
n_samples = len(dataset)

#dataset = torch.Tensor(dataset,dtype=torch.double)   #second place
#net.float()                                          #thrid place

net.forward(dataset[0])         #fourth place
```
In the file data1.txt is a csv formatted text file containing certain numbers, and each dataset[i] is a size 100 by 1 torch.Tensor object of dtype torch.float64. The error message is as follows:
```
Traceback (most recent call last):
  File "Z:\Wrong.py", line 33, in <module>
    net.forward(dataset[0])
  File "Z:\Wrong.py", line 23, in forward
    x = F.relu(self.layer(x))
  File "E:\Python38\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "E:\Python38\lib\site-packages\torch\nn\modules\linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "E:\Python38\lib\site-packages\torch\nn\functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'mat2' in call to _th_mm
```
It seems that I should change the dtype of the numbers in dataset to torch.double. I tried things like changing the line at the first place to self.data = torch.tensor(np.loadtxt(file_name,delimiter = ','),dtype=torch.double) changing the line at the fourth place to net.forward(dataset[0].double()) uncommenting one of the two lines at the second or the thrid place I think these are the solutions I have seen from similar questions, but they either give new errors or don't do anything. What should I do? Update: So I got it working by changing the first place to
```
self.data = torch.from_numpy(np.loadtxt(file_name,delimiter = ',')).float()
```
which is weird because it is exactly the opposite of the error message. Is this a bug? I'd still like some explaining.

Response: In short: your data has type double but your model has type float, this is not allowed in pytorch because only data with the same dtype can be fed into the model. In long: This issue is related to the default dtype of PyTorch and Numpy. I will first explain why this error happens and then suggest some solutions(but I think you will not need my solution once you understand the principle.) PyTorch has a couple of dtypes https://pytorch.org/docs/stable/tensors.html. Two of them are closely related to the issue you had: torch.float32(aka torch.float) torch.float64(aka torch.double) It's important to know the default dtype of PyTorch Tensors is torch.float32(aka torch.float). This means when you create a tensor, its default dtype is torch.float32.try: torch.ones(1).dtype . This will print torch.float32 in default case. And also the model's parameters are of this dtype by default. In your case, net = Net(100,1) will create a model whose dtype of parameters are torch.float32 Then we need to talk about Numpy: The default dtype of Numpy ndarray is numpy.float64. This means when you create a numpy array, its default dtype is numpy.float64.try: np.ones(1).dtype . This will print dtype('float64') in default case. In your case, your data come from a local file loaded by np.loadtxt, so the data is first loaded as dtype('float64')(as a numpy array) and then converted to a torch tensor of dtype torch.float64(aka torch.double). This is what happens when you convert a numpy array to torch tensor: they will have the corresponding dtype. I think now the issue is pretty clear, you have a model whose parameters are of torch.float32(aka torch.float) but tries to run it on data of torch.float64(aka torch.double). This is also what the error message tries to say:Expected object of scalar type Double but got scalar type Float for argument Solutions: You have alreay found one: convert your data to torch.float32 by calling tensor.float() You can also specify the dtype when load the data: np.loadtxt(file_name,delimiter = ',',dtype="float32")


Question: Multi label classification in pytorch I have a multi-label classification problem. I have 11 classes, around 4k examples. Each example can have from 1 to 4-5 label. At the moment, i'm training a classifier separately for each class with log_loss. As you can expect, it is taking quite some time to train 11 classifier, and i would like to try another approach and to train only 1 classifier. The idea is that the last layer of this classifer would have 11 nodes, and would output a real number by classes which would be converted to a proba by a sigmoid. The loss I want to optimize is the mean of the log_loss on all classes. Unfortunately, i'm some kind of noob with pytorch, and even by reading the source code of the losses, i can't figure out if one of the already existing losses does exactly what i want, or if I should create a new loss, and if that's the case, i don't really know how to do it. To be very specific, i want to give for each element of the batch one vector of size 11(which contains a real number for each label (the closer to infinity, the closer this class is predicted to be 1), and 1 vector of size 11 (which contains a 1 at every true label), and be able to compute the mean log_loss on all 11 labels, and optimize my classifier based on that loss. Any help would be greatly appreciated :)

Response:
You are looking for torch.nn.BCELoss. Here's example code:
```py
import torch

batch_size = 2
num_classes = 11

loss_fn = torch.nn.BCELoss()

outputs_before_sigmoid = torch.randn(batch_size, num_classes)
sigmoid_outputs = torch.sigmoid(outputs_before_sigmoid)
target_classes = torch.randint(0, 2, (batch_size, num_classes))  # randints in [0, 2).

loss = loss_fn(sigmoid_outputs, target_classes)

# alternatively, use BCE with logits, on outputs before sigmoid.
loss_fn_2 = torch.nn.BCEWithLogitsLoss()
loss2 = loss_fn_2(outputs_before_sigmoid, target_classes)
assert loss == loss2
```


Question:
img should be PIL Image. Got  I'm trying to iterate through a loader to check if it's working, however the below error is given: TypeError: img should be PIL Image. Got  I've tried adding both transforms.ToTensor() and transforms.ToPILImage() and it gives me an error asking for the opposite. i.e, with ToPILImage(), it will ask for tensor, and vice versa.
```
# Imports here
%matplotlib inline
import matplotlib.pyplot as plt
from torch import nn, optim
import torch.nn.functional as F
import torch
from torchvision import transforms, datasets, models
import seaborn as sns
import pandas as pd
import numpy as np

data_dir = 'flowers'
train_dir = data_dir + '/train'
valid_dir = data_dir + '/valid'
test_dir = data_dir + '/test'

#Creating transform for training set
train_transforms = transforms.Compose(
[transforms.Resize(255),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.RandomHorizontalFlip(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

#Creating transform for test set
test_transforms = transforms.Compose(
[transforms.Resize(255),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])])

#transforming for all data
train_data = datasets.ImageFolder(train_dir, transform=train_transforms)
test_data = datasets.ImageFolder(test_dir, transform = test_transforms)
valid_data = datasets.ImageFolder(valid_dir, transform = test_transforms)

#Creating data loaders for test and training sets
trainloader = torch.utils.data.DataLoader(train_data, batch_size = 32,
shuffle = True)
testloader = torch.utils.data.DataLoader(test_data, batch_size=32)
images, labels = next(iter(trainloader))
```
It should allow me to simply see the image once I run plt.imshow(images[0]), if its working correctly.

Response:
transforms.RandomHorizontalFlip() works on PIL.Images, not torch.Tensor. In your code above, you are applying transforms.ToTensor() prior to transforms.RandomHorizontalFlip(), which results in tensor. But, as per the official pytorch documentation here, transforms.RandomHorizontalFlip() horizontally flip the given PIL Image randomly with a given probability. So, just change the order of your transformation in above code, like below:
```
train_transforms = transforms.Compose([transforms.Resize(255),
                                       transforms.CenterCrop(224),
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])
```


Question:
Creating one hot vector from indices given as a tensor I have a tensor of size 4 x 6 where 4 is batch size and 6 is sequence length. Every element of the sequence vectors are some index (0 to n). I want to create a 4 x 6 x n tensor where the vectors in 3rd dimension will be one hot encoding of the index which means I want to put 1 in the specified index and rest of the values will be zero. For example, I have the following tensor:
```
[[5, 3, 2, 11, 15, 15],
[1, 4, 6, 7, 3, 3],
[2, 4, 7, 8, 9, 10],
[11, 12, 15, 2, 5, 7]]
```
Here, all the values are in between (0 to n) where n = 15. So, I want to convert the tensor to a 4 X 6 X 16 tensor where the third dimension will represent one hot encoding vector. How can I do that using PyTorch functionalities? Right now, I am doing this with loop but I want to avoid looping!

Response:
NEW ANSWER As of PyTorch 1.1, there is a one_hot function in torch.nn.functional. Given any tensor of indices indices and a maximal index n, you can create a one_hot version as follows:
```
n = 5
indices = torch.randint(0,n, size=(4,7))
one_hot = torch.nn.functional.one_hot(indices, n) # size=(4,7,n)
```
Very old Answer At the moment, slicing and indexing can be a bit of a pain in PyTorch from my experience. I assume you don't want to convert your tensors to numpy arrays. The most elegant way I can think of at the moment is to use sparse tensors and then convert to a dense tensor. That would work as follows:
```
from torch.sparse import FloatTensor as STensor

batch_size = 4
seq_length = 6
feat_dim = 16

batch_idx = torch.LongTensor([i for i in range(batch_size) for s in range(seq_length)])
seq_idx = torch.LongTensor(list(range(seq_length))*batch_size)
feat_idx = torch.LongTensor([[5, 3, 2, 11, 15, 15], [1, 4, 6, 7, 3, 3],
                             [2, 4, 7, 8, 9, 10], [11, 12, 15, 2, 5, 7]]).view(24,)

my_stack = torch.stack([batch_idx, seq_idx, feat_idx]) # indices must be nDim * nEntries
my_final_array = STensor(my_stack, torch.ones(batch_size * seq_length),
                         torch.Size([batch_size, seq_length, feat_dim])).to_dense()

print(my_final_array)
```
Note: PyTorch is undergoing some work currently, that will add numpy style broadcasting and other functionalities within the next two or three weeks and other functionalities. So it's possible, there'll be better solutions available in the near future. Hope this helps you a bit.


Question: How to multiply a tensor row-wise by a vector in PyTorch? When I have a tensor m of shape [12, 10] and a vector s of scalars with shape [12], how can I multiply each row of m with the corresponding scalar in s?

Response:
You need to add a corresponding singleton dimension:
```
m * s[:, None]
```
s[:, None] has size of (12, 1) when multiplying a (12, 10) tensor by a (12, 1) tensor pytorch knows to broadcast s along the second singleton dimension and perform the "element-wise" product correctly.


Question:
Calling the forward method in PyTorch vs. calling the model instance A lot of the PyTorch tutorials I've been viewing do something like this. Define model:
```
class Network(nn.Module):
    def __init__():
        super().__init__()
        self.conv1 = ..
        ...

    def forward(x)
        ...
    ...
```
Once the Network has been instantiated (net = Network()), the people in the tutorials write net(input_data) instead of net.forward(input_data). I tried net.forward() and it gives the same results as net(). Why is this a common practice, and also why does this work?

Response: You should avoid calling Module.forward. The difference is that all the hooks are dispatched in the __call__ function see this, so if you call .forward and have hooks in your model, the hooks won’t have any effect. Inshort when you call Module.forward, pytorch hooks wont have any effect Detailed answer can be found in this post


Question:
Output and Broadcast shape mismatch in MNIST, torchvision I am getting following error when using MNIST dataset in Torchvision
```
RuntimeError: output with shape [1, 28, 28] doesn't match the broadcast shape [3, 28, 28]
```
Here is my code:
```
import torch
from torchvision import datasets, transforms

transform = transforms.Compose([transforms.ToTensor(),
                            transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                          ])
trainset = datasets.MNIST('~/.pytorch/MNIST_data/', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)
images, labels = next(iter(trainloader))
```

Response:
The error is due to color vs grayscale on the dataset, the dataset is grayscale. I fixed it by changing transform to
```
transform = transforms.Compose([transforms.ToTensor(),
  transforms.Normalize((0.5,), (0.5,))
])
```


Question:
Indexing a tensor with None in PyTorch I've seen this syntax to index a tensor in PyTorch, not sure what it means:
```
v = torch.div(t, n[:, None])
```
where v, t, and n are tensors. What is the role of "None" here? I can't seem to find it in the documentation.

Response:
Similar to NumPy you can insert a singleton dimension ("unsqueeze" a dimension) by indexing this dimension with None. In turn n[:, None] will have the effect of inserting a new dimension on dim=1. This is equivalent to n.unsqueeze(dim=1):
```

```python
n = torch.rand(3, 100, 100)
```

```python
n[:, None].shape
#Output
#(3, 1, 100, 100)
```

```python
n.unsqueeze(1).shape
#Output
#(3, 1, 100, 100)
#```
#Here are some other types of None indexings. In the example above : is was used as a placeholder to designate the first dimension dim=0. If you want to insert a dimension on dim=2, you can add a second : as n[:, :, None]. You can also place None with respect to the last dimension instead. To do so you can use the ellipsis syntax ...: n[..., None] will insert a dimension last, i.e. n.unsqueeze(dim=-1). n[..., None, :] on the before last dimension, i.e. n.unsqueeze(dim=-2).
```


Question: Difference between autograd.grad and autograd.backward? Suppose I have my custom loss function and I want to fit the solution of some differential equation with help of my neural network. So in each forward pass, I am calculating the output of my neural net and then calculating the loss by taking the MSE with the expected equation to which I want to fit my perceptron. Now my doubt is: should I use grad(loss) or should I do loss.backward() for backpropagation to calculate and update my gradients? I understand that while using loss.backward() I have to wrap my tensors with Variable and have to set the requires_grad = True for the variables w.r.t which I want to take the gradient of my loss. So my questions are : Does grad(loss) also requires any such explicit parameter to identify the variables for gradient computation? How does it actually compute the gradients? Which approach is better? what is the main difference between the two in a practical scenario. It would be better if you could explain the practical implications of both approaches because whenever I try to find it online I am just bombarded with a lot of stuff that isn't much relevant to my project.

Response:
TLDR; Both are different interfaces for gradient computation: torch.autograd.grad is non-mutable while torch.autograd.backward is. Descriptions The torch.autograd module is the automatic differentiation package for PyTorch. As described in the documentation it only requires minimal change to the code base to be used: you only need to declare Tensors for which gradients should be computed with the requires_grad=True keyword. The two main functions torch.autograd provides for gradient computation are torch.autograd.backward and torch.autograd.grad: torch.autograd.backward (source) torch.autograd.grad (source) Description Computes the sum of gradients of given tensors with respect to graph leaves. Computes and returns the sum of gradients of outputs with respect to the inputs. Header torch.autograd.backward( tensors, grad_tensors=None, retain_graph=None, create_graph=False, grad_variables=None, inputs=None) torch.autograd.grad( outputs, inputs, grad_outputs=None, retain_graph=None, create_graph=False, only_inputs=True, allow_unused=False) Parameters - tensors – Tensors of which the derivative will be computed.- grad_tensors – The "vector" in the Jacobian-vector product, usually gradients w.r.t. each element of corresponding tensors.- retain_graph – If False, the graph used to compute the grad will be freed. [...] - inputs – Inputs w.r.t. which the gradient be will be accumulated into .grad. All other Tensors will be ignored. If not provided, the gradient is accumulated into all the leaf Tensors that were used [...]. - outputs – outputs of the differentiated function.- inputs – Inputs w.r.t. which the gradient will be returned (and not accumulated into .grad).- grad_outputs – The "vector" in the Jacobian-vector product, usually gradients w.r.t. each element of corresponding tensors.- retain_graph – If False, the graph used to compute the grad will be freed. [...]. Usage examples In terms of high-level usage, you can look at torch.autograd.grad as a non-mutable function. As mentioned in the documentation table above, it will not accumulate the gradients on the grad attribute but instead return the computed partial derivatives. In contrast torch.autograd.backward will be able to mutate the tensors by updating the grad attribute of leaf nodes, the function won't return any value. In other words, the latter is more suitable when computing gradients for a large number of parameters. In the following, we will take two inputs (x1 and, x2), calculate a tensor y with them, and then compute the partial derivatives of the result w.r.t both inputs, i.e. dL/dx1 and dL/dx2:
```

```python
x1 = torch.rand(1, requires_grad=True)
```

```python
x2 = torch.rand(1, requires_grad=True)
```

```python
x1, x2
#Output
#(tensor(0.3939, grad_fn=<UnbindBackward>),
# tensor(0.7965, grad_fn=<UnbindBackward>))
#```
#Inference:
#```
```

```python
y = x1**2 + 5*x2
```

```python
y
#Output
#tensor(4.1377, grad_fn=<AddBackward0>)
#```
#Since y was computed using tensor(s) requiring gradients (i.e. with requires_grad=True) - *outside of a torch.no_grad context. It will have a grad_fn function attached. This callback is used to backpropagate onto the computation graph to compute the gradients of preceding tensor nodes. torch.autograd.grad: Here we provide torch.ones_like(y) as the grad_outputs.
#```
```

```python
torch.autograd.grad(y, (x1, x2), torch.ones_like(y))
#Output
#(tensor(0.7879), tensor(5.))
#```
#The above output is a tuple containing the two partial derivatives w.r.t. to the provided inputs respectively in order of appearance, i.e. dL/dx1 and dL/dx2. This corresponds to the following computation:
#```
## dL/dx1 = dL/dy * dy/dx1 = grad_outputs @ 2*x1
## dL/dx2 = dL/dy * dy/dx2 = grad_outputs @ 5
#```
#torch.autograd.backward: in contrast, it will mutate the provided tensors by updating the grad of the tensors that have been used to compute the output tensor and that require gradients. It is equivalent to the torch.Tensor.backward API. Here, we go through the same example by defining x1, x2, and y again. We call backward:
#```
```

```python
# y.backward(torch.ones_like(y))
```

```python
torch.autograd.backward(y, torch.ones_like(y))
#Output
#None
#```
#Then you can retrieve the gradients on x1.grad and x2.grad:
#```
```

```python
x1.grad, x2.grad
#Output
#(tensor(0.7879), tensor(5.))
#```
#In conclusion: both perform the same operation. They are two different interfaces to interact with the autograd library and perform gradient computations. The latter, torch.autograd.backward (equivalent to torch.Tensor.backward), is generally used in neural networks training loops to compute the partial derivative of the loss w.r.t each one of the model's parameters. You can read more about how torch.autograd.grad works by reading through this other answer I made on: Meaning of grad_outputs in PyTorch's torch.autograd.grad.
```


Question: what's the difference between torch.Tensor() vs torch.empty() in pytorch? I have tried it out as below. It seems to me they're the same. What's the difference between torch.Tensor() vs torch.empty() in pytorch?

Response:
torch.Tensor() is just an alias to torch.FloatTensor() which is the default type of tensor, when no dtype is specified during tensor construction. From the torch for numpy users notes, it seems that torch.Tensor() is a drop-in replacement of numpy.empty() So, in essence torch.FloatTensor() and torch.empty() does the same job of returning a tensor filled with garbage values of dtype torch.float32. Below is a small run:
```

```python
torch.FloatTensor(2, 3)
#Output
#tensor([[-1.0049e+08,  4.5688e-41, -8.9389e-38],
#        [ 3.0638e-41,  4.4842e-44,  0.0000e+00]])
```

```python
torch.FloatTensor(2, 3)
#Output
#tensor([[-1.0049e+08,  4.5688e-41, -1.6512e-38],
#        [ 3.0638e-41,  4.4842e-44,  0.0000e+00]])
#```
#```
```

```python
torch.empty(2, 3)
#Output
#tensor([[-1.0049e+08,  4.5688e-41, -9.0400e-38],
#        [ 3.0638e-41,  4.4842e-44,  0.0000e+00]])
```

```python
torch.empty(2, 3)
#Output
#tensor([[-1.0049e+08,  4.5688e-41, -9.2852e-38],
#        [ 3.0638e-41,  4.4842e-44,  0.0000e+00]])
#```
```


Question:
PyTorch CUDA vs Numpy for arithmetic operations? Fastest? I performed element-wise multiplication using Torch with GPU support and Numpy using the functions below and found that Numpy loops faster than Torch which shouldn't be the case, I doubt. I want to know how to perform general arithmetic operations with Torch using GPU. Note: I ran these code snippets in Google Colab notebook Define the default tensor type to enable global GPU flag
```
torch.set_default_tensor_type(torch.cuda.FloatTensor if
                              torch.cuda.is_available() else
                              torch.FloatTensor)
```
Initialize Torch variables
```
x = torch.Tensor(200, 100)  # Is FloatTensor
y = torch.Tensor(200,100)
```
Function in question
```
def mul(d,f):
    g = torch.mul(d,f).cuda()  # I explicitly called cuda() which is not necessary
    return g
```
When call the function above as %timeit mul(x,y) Returns: The slowest run took 10.22 times longer than the fastest. This could mean hat an intermediate result is being cached. 10000 loops, best of 3: 50.1 µs per loop Now trial with numpy, Used the same values from torch variables
```
x_ = x.data.cpu().numpy()
y_ = y.data.cpu().numpy()
```
```
def mul_(d,f):
    g = d*f
    return g
```
%timeit mul_(x_,y_) Returns The slowest run took 12.10 times longer than the fastest. This could mean that an intermediate result is being cached. 100000 loops, best of 3: 7.73 µs per loop Needs some help to understand GPU enabled Torch operations.

Response: GPU operations have to additionally get memory to/from the GPU The problem is that your GPU operation always has to put the input on the GPU memory, and then retrieve the results from there, which is a quite costly operation. NumPy, on the other hand, directly processes the data from the CPU/main memory, so there is almost no delay here. Additionally, your matrices are extremely small, so even in the best-case scenario, there should only be a minute difference. This is also partially the reason why you use mini-batches when training on a GPU in neural networks: Instead of having several extremely small operations, you now have "one big bulk" of numbers that you can process in parallel. Also note that GPU clock speeds are generally way lower than CPU clocks, so the GPU only really shines because it has way more cores. If your matrix does not utilize all of them fully, you are also likely to see a faster result on your CPU. TL;DR: If your matrix is big enough, you will eventually see a speed-up in CUDA than Numpy, even with the additional cost of the GPU transfer.


Question:
What does the underscore suffix in PyTorch functions mean? In PyTorch, many methods of a tensor exist in two versions - one with a underscore suffix, and one without. If I try them out, they seem to do the same thing:
```

```python
import torch
```

```python
a = torch.tensor([2, 4, 6])
```

```python
a.add(10)
#Output
#tensor([12, 14, 16])
```

```python
a.add_(10)
#Output
#tensor([12, 14, 16])
#```
#What is the difference between torch.add and torch.add_ torch.sub and torch.sub_ ...and so on?
```

Response:
You have already answered your own question that the underscore indicates in-place operations in PyTorch. However I want to point out briefly why in-place operations can be problematic: First of all on the PyTorch sites it is recommended to not use in-place operations in most cases. Unless working under heavy memory pressure it is more efficient in most cases to not use in-place operations. https://pytorch.org/docs/stable/notes/autograd.html#in-place-operations-with-autograd Secondly there can be problems calculating the gradients when using in-place operations: Every tensor keeps a version counter, that is incremented every time it is marked dirty in any operation. When a Function saves any tensors for backward, a version counter of their containing Tensor is saved as well. Once you access self.saved_tensors it is checked, and if it is greater than the saved value an error is raised. This ensures that if you’re using in-place functions and not seeing any errors, you can be sure that the computed gradients are correct. Same source as above. Here is a shot and slightly modified example taken from the answer you've posted: First the in-place version:
```
import torch
a = torch.tensor([2, 4, 6], requires_grad=True, dtype=torch.float)
adding_tensor = torch.rand(3)
b = a.add_(adding_tensor)
c = torch.sum(b)
c.backward()
print(c.grad_fn)
```
Which leads to this error:
```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-27-c38b252ffe5f> in <module>
      2 a = torch.tensor([2, 4, 6], requires_grad=True, dtype=torch.float)
      3 adding_tensor = torch.rand(3)
----> 4 b = a.add_(adding_tensor)
      5 c = torch.sum(b)
      6 c.backward()

RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.
```
Secondly the non in-place version:
```
import torch
a = torch.tensor([2, 4, 6], requires_grad=True, dtype=torch.float)
adding_tensor = torch.rand(3)
b = a.add(adding_tensor)
c = torch.sum(b)
c.backward()
print(c.grad_fn)
```
Which works just fine - output:
```
<SumBackward0 object at 0x7f06b27a1da0>
```
So as a take-away I just wanted to point out to carefully use in-place operations in PyTorch.


Question: How to use autograd.gradcheck in PyTorch? The documentation does not include any example use case of gradcheck, where would it be useful?

Response:
There's an example use case provided in the documentation here: https://pytorch.org/docs/master/notes/extending.html You probably want to check if the backward method you implemented actually computes the derivatives of your function. It is possible by comparing with numerical approximations using small finite differences:
```py
from torch.autograd import gradcheck

# gradcheck takes a tuple of tensors as input, check if your gradient
# evaluated with these tensors are close enough to numerical
# approximations and returns True if they all verify this condition.
input = (torch.randn(20,20,dtype=torch.double,requires_grad=True), torch.randn(30,20,dtype=torch.double,requires_grad=True))
test = gradcheck(linear, input, eps=1e-6, atol=1e-4)
print(test)
```
As the quote above suggests, the purpose of the gradcheck function is to verify that a custom backward function agrees with a numerical approximation of the gradient. The primary use case is when you're implementing a custom backward operation. In very few cases should you be implementing your own backward function in PyTorch. This is because PyTorch's autograd functionality takes care of computing gradients for the vast majority of operations. The most obvious exceptions are You have a function which can't be expressed as a finite combination of other differentiable functions (for example, if you needed the incomplete gamma function, you might want to write your own forward and backward which used numpy and/or lookup tables). You're looking to speed up the computation of a particularly complicated expression for which the gradient could be drastically simplified after applying the chain rule.


Question:
Bidirectional LSTM output question in PyTorch Hi I have a question about how to collect the correct result from a BI-LSTM module’s output. Suppose I have a 10-length sequence feeding into a single-layer LSTM module with 100 hidden units:
```
lstm = nn.LSTM(5, 100, 1, bidirectional=True)
```
output will be of shape:
```
[10 (seq_length), 1 (batch),  200 (num_directions * hidden_size)]
# or according to the doc, can be viewed as
[10 (seq_length), 1 (batch),  2 (num_directions), 100 (hidden_size)]
```
If I want to get the 3rd (1-index) input’s output at both directions (two 100-dim vectors), how can I do it correctly? I know output[2, 0] will give me a 200-dim vector. Does this 200 dim vector represent the output of 3rd input at both directions? A thing bothering me is that when do reverse feeding, the 3rd (1-index) output vector is calculated from the 8th(1-index) input, right? Will pytorch automatically take care of this and group output considering direction? Thanks!

Response:
Yes, when using a BiLSTM the hidden states of the directions are just concatenated (the second part after the middle is the hidden state for feeding in the reversed sequence). So splitting up in the middle works just fine. As reshaping works from the right to the left dimensions you won't have any problems in separating the two directions. Here is a small example:
```py
# so these are your original hidden states for each direction
# in this case hidden size is 5, but this works for any size
direction_one_out = torch.tensor(range(5))
direction_two_out = torch.tensor(list(reversed(range(5))))
print('Direction one:')
print(direction_one_out)
print('Direction two:')
print(direction_two_out)

# before outputting they will be concatinated
# I'm adding here batch dimension and sequence length, in this case seq length is 1
hidden = torch.cat((direction_one_out, direction_two_out), dim=0).view(1, 1, -1)
print('\nYour hidden output:')
print(hidden, hidden.shape)

# trivial case, reshaping for one hidden state
hidden_reshaped = hidden.view(1, 1, 2, -1)
print('\nReshaped:')
print(hidden_reshaped, hidden_reshaped.shape)

# This works as well for abitrary sequence lengths as you can see here
# I've set sequence length here to 5, but this will work for any other value as well
print('\nThis also works for more multiple hidden states in a tensor:')
multi_hidden = hidden.expand(5, 1, 10)
print(multi_hidden, multi_hidden.shape)
print('Directions can be split up just like this:')
multi_hidden = multi_hidden.view(5, 1, 2, 5)
print(multi_hidden, multi_hidden.shape)
```
Output:
```py
Direction one:
tensor([0, 1, 2, 3, 4])
Direction two:
tensor([4, 3, 2, 1, 0])

Your hidden output:
tensor([[[0, 1, 2, 3, 4, 4, 3, 2, 1, 0]]]) torch.Size([1, 1, 10])

Reshaped:
tensor([[[[0, 1, 2, 3, 4],
          [4, 3, 2, 1, 0]]]]) torch.Size([1, 1, 2, 5])

This also works for more multiple hidden states in a tensor:
tensor([[[0, 1, 2, 3, 4, 4, 3, 2, 1, 0]],

        [[0, 1, 2, 3, 4, 4, 3, 2, 1, 0]],

        [[0, 1, 2, 3, 4, 4, 3, 2, 1, 0]],

        [[0, 1, 2, 3, 4, 4, 3, 2, 1, 0]],

        [[0, 1, 2, 3, 4, 4, 3, 2, 1, 0]]]) torch.Size([5, 1, 10])
Directions can be split up just like this:
tensor([[[[0, 1, 2, 3, 4],
          [4, 3, 2, 1, 0]]],

        [[[0, 1, 2, 3, 4],
          [4, 3, 2, 1, 0]]],

        [[[0, 1, 2, 3, 4],
          [4, 3, 2, 1, 0]]],

        [[[0, 1, 2, 3, 4],
          [4, 3, 2, 1, 0]]],

        [[[0, 1, 2, 3, 4],
          [4, 3, 2, 1, 0]]]]) torch.Size([5, 1, 2, 5])
```
Hope this helps! :)


Question: BertForSequenceClassification vs. BertForMultipleChoice for sentence multi-class classification I'm working on a text classification problem (e.g. sentiment analysis), where I need to classify a text string into one of five classes. I just started using the Huggingface Transformer package and BERT with PyTorch. What I need is a classifier with a softmax layer on top so that I can do 5-way classification. Confusingly, there seem to be two relevant options in the Transformer package: BertForSequenceClassification and BertForMultipleChoice. Which one should I use for my 5-way classification task? What are the appropriate use cases for them? The documentation for BertForSequenceClassification doesn't mention softmax at all, although it does mention cross-entropy. I am not sure if this class is only for 2-class classification (i.e. logistic regression). Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. labels (torch.LongTensor of shape (batch_size,), optional, defaults to None) – Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy). The documentation for BertForMultipleChoice mentions softmax, but the way the labels are described, it sound like this class is for multi-label classification (that is, a binary classification for multiple labels). Bert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. labels (torch.LongTensor of shape (batch_size,), optional, defaults to None) – Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices] where num_choices is the size of the second dimension of the input tensors. Thank you for any help.

Response: The answer to this lies in the (admittedly very brief) description of what the tasks are about: [BertForMultipleChoice] [...], e.g. for RocStories/SWAG tasks. When looking at the paper for SWAG, it seems that the task is actually learning to choose from varying options. This is in contrast to your "classical" classification task, in which the "choices" (i.e., classes) do not vary across your samples, which is exactly what BertForSequenceClassification is for. Both variants can in fact be for an arbitrary number of classes (in the case of BertForSequenceClassification), respectively choices (for BertForMultipleChoice), via changing the labels parameter in the config. But, since it seems like you are dealing with a case of "classical classification", I suggest using the BertForSequenceClassification model. Shortly addressing the missing Softmax in BertForSequenceClassification: Since classification tasks can compute loss across classes indipendent of the sample (unlike multiple choice, where your distribution is changing), this allows you to use Cross-Entropy Loss, which factors in Softmax in the backpropagation step for increased numerical stability.


Question:
How to iterate over two dataloaders simultaneously using pytorch? I am trying to implement a Siamese network that takes in two images. I load these images and create two separate dataloaders. In my loop I want to go through both dataloaders simultaneously so that I can train the network on both images.
```
for i, data in enumerate(zip(dataloaders1, dataloaders2)):

    # get the inputs
    inputs1 = data[0][0].cuda(async=True);
    labels1 = data[0][1].cuda(async=True);

    inputs2 = data[1][0].cuda(async=True);
    labels2 = data[1][1].cuda(async=True);

    labels1 = labels1.view(batchSize,1)
    labels2 = labels2.view(batchSize,1)

    # zero the parameter gradients
    optimizer.zero_grad()

    # forward + backward + optimize
    outputs1 = alexnet(inputs1)
    outputs2 = alexnet(inputs2)
```
The return value of the dataloader is a tuple. However, when I try to use zip to iterate over them, I get the following error:
```
OSError: [Errno 24] Too many open files
Exception NameError: "global name 'FileNotFoundError' is not defined" in <bound method _DataLoaderIter.__del__ of <torch.utils.data.dataloader._DataLoaderIter object at 0x7f2d3c00c190>> ignored
```
Shouldn't zip work on all iterable items? But it seems like here I can't use it on dataloaders. Is there any other way to pursue this? Or am I approaching the implementation of a Siamese network incorrectly?

Response:
Further to what it is already mentioned, cycle() and zip() might create a memory leakage problem - especially when using image datasets! To solve that, instead of iterating like this:
```
dataloaders1 = DataLoader(DummyDataset(0, 100), batch_size=10, shuffle=True)
dataloaders2 = DataLoader(DummyDataset(0, 200), batch_size=10, shuffle=True)
num_epochs = 10

for epoch in range(num_epochs):

    for i, (data1, data2) in enumerate(zip(cycle(dataloaders1), dataloaders2)):

        do_cool_things()
```
you could use:
```
dataloaders1 = DataLoader(DummyDataset(0, 100), batch_size=10, shuffle=True)
dataloaders2 = DataLoader(DummyDataset(0, 200), batch_size=10, shuffle=True)
num_epochs = 10

for epoch in range(num_epochs):
    dataloader_iterator = iter(dataloaders1)

    for i, data1 in enumerate(dataloaders2)):

        try:
            data2 = next(dataloader_iterator)
        except StopIteration:
            dataloader_iterator = iter(dataloaders1)
            data2 = next(dataloader_iterator)

        do_cool_things()
```
Bear in mind that if you use labels as well, you should replace in this example data1 with (inputs1,targets1) and data2 with inputs2,targets2, as @Sajad Norouzi said. KUDOS to this one: https://github.com/pytorch/pytorch/issues/1917#issuecomment-433698337


Question:
How to disable progress bar in Pytorch Lightning I have a lot of issues withe the tqdm progress bar in Pytorch Lightning: when I run trainings in a terminal, the progress bars overwrite themselves. At the end of an training epoch, a validation progress bar is printed under the training bar, but when that ends, the progress bar from the next training epoch is printed over the one from the previous epoch. hence it's not possible to see the losses from previous epochs.
```
INFO:root:  Name    Type Params
0   l1  Linear    7 K
Epoch 2:  56%|████████████▊          | 2093/3750 [00:05<00:03, 525.47batch/s, batch_nb=1874, loss=0.714, training_loss=0.4, v_nb=51]
```
the progress bars wobbles from left to right, caused by the changing in number of digits behind the decimal point of some losses. when running in Pycharm, the validation progress bar is not printed, but in stead,
```
INFO:root:  Name    Type Params
0   l1  Linear    7 K
Epoch 1:  50%|█████     | 1875/3750 [00:05<00:05, 322.34batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  50%|█████     | 1879/3750 [00:05<00:05, 319.41batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  52%|█████▏    | 1942/3750 [00:05<00:04, 374.05batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  53%|█████▎    | 2005/3750 [00:05<00:04, 425.01batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  55%|█████▌    | 2068/3750 [00:05<00:03, 470.56batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  57%|█████▋    | 2131/3750 [00:05<00:03, 507.69batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  59%|█████▊    | 2194/3750 [00:06<00:02, 538.19batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  60%|██████    | 2257/3750 [00:06<00:02, 561.20batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  62%|██████▏   | 2320/3750 [00:06<00:02, 579.22batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  64%|██████▎   | 2383/3750 [00:06<00:02, 591.58batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  65%|██████▌   | 2445/3750 [00:06<00:02, 599.77batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  67%|██████▋   | 2507/3750 [00:06<00:02, 605.00batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  69%|██████▊   | 2569/3750 [00:06<00:01, 607.04batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  70%|███████   | 2633/3750 [00:06<00:01, 613.98batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
```
I would like to know if these issues can be solved or else how can I disable the progress bar and instead, just print some log details on the screen.

Response:
F.Y.I. show_progress_bar=False deprecated since version 0.7.2, but you can use progress_bar_refresh_rate=0 update: progress_bar_refresh_rate has been deprecated in v1.5 and will be removed in v1.7. To disable the progress bar, set enable_progress_bar to false
```
progress_bar_refresh_rate: How often to refresh progress bar (in steps). Value ``0`` disables progress bar.
    Ignored when a custom progress bar is passed to :paramref:`~Trainer.callbacks`. Default: None, means
    a suitable value will be chosen based on the environment (terminal, Google COLAB, etc.).

    .. deprecated:: v1.5
        ``progress_bar_refresh_rate`` has been deprecated in v1.5 and will be removed in v1.7.
        Please pass :class:`~pytorch_lightning.callbacks.progress.TQDMProgressBar` with ``refresh_rate``
        directly to the Trainer's ``callbacks`` argument instead. To disable the progress bar,
        pass ``enable_progress_bar = False`` to the Trainer.

enable_progress_bar: Whether to enable to progress bar by default.
```


Question: How to convert one-hot vector to label index and back in Pytorch? How to transform vectors of labels to one-hot encoding and back in Pytorch? The solution to the question was copied to here after having to go through the entire forum discussion, instead of just finding an easy one from googling.

Response:
From the Pytorch forums
```
import torch
import numpy as np

labels = torch.randint(0, 10, (10,))

# labels --> one-hot
one_hot = torch.nn.functional.one_hot(labels)
# one-hot --> labels
labels_again = torch.argmax(one_hot, dim=1)

np.testing.assert_equals(labels.numpy(), labels_again.numpy())
```


Question:
How to convert a tensor of booleans to ints in PyTorch? Suppose, we have a tensor
```
t = torch.tensor([True, False, True, False])
```
How do we convert it to an integer tensor with values [1, 0, 1, 0]?

Response:
The solution is just a single line of code. To convert a tensor t with values [True, False, True, False] to an integer tensor, just do the following.
```
t = torch.tensor([True, False, True, False])
t_integer = t.long()
print(t_integer)
[1, 0, 1, 0]
```


Question:
Difficulty in understanding the tokenizer used in Roberta model
```py
from transformers import AutoModel, AutoTokenizer

tokenizer1 = AutoTokenizer.from_pretrained("roberta-base")
tokenizer2 = AutoTokenizer.from_pretrained("bert-base-cased")

sequence = "A Titan RTX has 24GB of VRAM"
print(tokenizer1.tokenize(sequence))
print(tokenizer2.tokenize(sequence))
```
Output: ['A', 'ĠTitan', 'ĠRTX', 'Ġhas', 'Ġ24', 'GB', 'Ġof', 'ĠVR', 'AM'] ['A', 'Titan', 'R', '##T', '##X', 'has', '24', '##GB', 'of', 'V', '##RA', '##M'] Bert model uses WordPiece tokenizer. Any word that does not occur in the WordPiece vocabulary is broken down into sub-words greedily. For example, 'RTX' is broken into 'R', '##T' and '##X' where ## indicates it is a subtoken. Roberta uses BPE tokenizer but I'm unable to understand a) how BPE tokenizer works? b) what does G represents in each of tokens?

Response: This question is extremely broad, so I'm trying to give an answer that focuses on the main problem at hand. If you feel the need to have other questions answered, please open another question focusing on one question at a time, see the [help/on-topic] rules for Stackoverflow. Essentially, as you've correctly identified, BPE is central to any tokenization in modern deep networks. I highly recommend you to read the original BPE paper by Sennrich et al., in which they also highlight a bit more of the history of BPEs. In any case, the tokenizers for any of the huggingface models are pretrained, meaning that they are usually generated from the training set of the algorithm beforehand. Common implementations such as SentencePiece also give a bit better understanding of it, but essentially the task is framed as a constrained optimization problem, where you specify a maximum number of k allowed vocabulary words (the constraint), and the algorithm tries to then keep as many words intact without exceeding k. if there are not enough words to cover the whole vocabulary, smaller units are used to approximate the vocabulary, which results in the splits observed in the example you gave. RoBERTa uses a variant called "byte-level BPE", the best explanation is probably given in this study by Wang et al.. The main benefit is, that it results in a smaller vocabulary while maintaining the quality of splits, from what I understand. The second part of your question is easier to explain; while BERT highlights the merging of two subsequent tokens (with ##), RoBERTa's tokenizer instead highlights the start of a new token with a specific unicode character (in this case, \u0120, the G with a dot). The best reason I could find for this was this thread, which argues that it basically avoids the use of whitespaces in training.


Question:
pytorch skip connection in a sequential model I am trying to wrap my head around skip connections in a sequential model. With the functional API I would be doing something as easy as (quick example, maybe not be 100% syntactically correct but should get the idea):
```python
x1 = self.conv1(inp)
x = self.conv2(x)
x = self.conv3(x)
x = self.conv4(x)

x = self.deconv4(x)
x = self.deconv3(x)
x = self.deconv2(x)
x = torch.cat((x, x1), 1))
x = self.deconv1(x)
```
I am now using a sequential model and trying to do something similar, create a skip connection that brings the activations of the first conv layer all the way to the last convTranspose. I have taken a look at the U-net architecture implemented here and it's a bit confusing, it does something like this:
```python
upconv = nn.ConvTranspose2d(inner_nc * 2, outer_nc,
                                    kernel_size=4, stride=2,
                                    padding=1, bias=use_bias)
down = [downrelu, downconv, downnorm]
up = [uprelu, upconv, upnorm]

if use_dropout:
    model = down + [submodule] + up + [nn.Dropout(0.5)]
else:
    model = down + [submodule] + up
```
Isn't this just adding layers to the sequential model well, sequentially? There is the down conv which is followed by submodule (which recursively adds inner layers) and then concatenated to up which is the upconv layer. I am probably missing something important on how the Sequential API works, but how does the code snipped from U-NET actually implements the skip?

Response:
Your observations are correct, but you may have missed the definition of UnetSkipConnectionBlock.forward() (UnetSkipConnectionBlock being the Module defining the U-Net block you shared), which may clarify this implementation: (from pytorch-CycleGAN-and-pix2pix/models/networks.py#L259)
```
# Defines the submodule with skip connection.
# X -------------------identity---------------------- X
#   |-- downsampling -- |submodule| -- upsampling --|
class UnetSkipConnectionBlock(nn.Module):

    # ...

    def forward(self, x):
        if self.outermost:
            return self.model(x)
        else:
            return torch.cat([x, self.model(x)], 1)
```
The last line is the key (applied for all inner blocks). The skip layer is simply done by concatenating the input x and the (recursive) block output self.model(x), with self.model the list of operations you mentioned -- so not so differently from the Functional code you wrote.


Question:
Is there any pytorch function can combine the specific continuous dimensions of tensor into one? Let's call the function I'm looking for "magic_combine", which can combine the continuous dimensions of tensor I give to it. For more specific, I want it to do the following thing:
```
a = torch.zeros(1, 2, 3, 4, 5, 6)
b = a.magic_combine(2, 5) # combine dimension 2, 3, 4
print(b.size()) # should be (1, 2, 60, 6)
```
I know that torch.view() can do the similar thing. But I'm just wondering if there is any more elegant way to achieve the goal?

Response:
There is a variant of flatten that takes start_dim and end_dim parameters. You can call it in the same way as your magic_combine (except that end_dim is inclusive).
```py
a = torch.zeros(1, 2, 3, 4, 5, 6)
b = a.flatten(2, 4) # combine dimension 2, 3, 4
print(b.size()) # should be (1, 2, 60, 6)
```
https://pytorch.org/docs/stable/generated/torch.flatten.html There is also a corresponding unflatten, in which you can specify a dimension to unflatten and a shape to unflatten it to.


Question:
AttributeError: 'torch.return_types.max' object has no attribute 'dim' - Maxpooling Channel I'm trying to do maxpooling over channel dimension:
```
class ChannelPool(nn.Module):
    def forward(self, input):
        return torch.max(input, dim=1)
```
but I get the error
```
AttributeError: 'torch.return_types.max' object has no attribute 'dim'
```

Response:
The torch.max function called with dim returns a tuple so:
```
class ChannelPool(nn.Module):
    def forward(self, input):
        input_max, max_indices = torch.max(input, dim=1)
        return input_max
```
From the documentation of torch.max: Returns a namedtuple (values, indices) where values is the maximum value of each row of the input tensor in the given dimension dim. And indices is the index location of each maximum value found (argmax).


Question:
Finding non-intersection of two pytorch tensors Thanks everyone in advance for your help! What I'm trying to do in PyTorch is something like numpy's setdiff1d. For example given the below two tensors:
```
t1 = torch.tensor([1, 9, 12, 5, 24]).to('cuda:0')
t2 = torch.tensor([1, 24]).to('cuda:0')
```
The expected output should be (sorted or unsorted):
```
torch.tensor([9, 12, 5])
```
Ideally the operations are done on GPU and no back and forth between GPU and CPU. Much appreciated!

Response:
I came across the same problem but the proposed solutions were far too slow when using larger arrays. The following simple solution works on CPU and GPU and is significantly faster than the other proposed solutions:
```
combined = torch.cat((t1, t2))
uniques, counts = combined.unique(return_counts=True)
difference = uniques[counts == 1]
intersection = uniques[counts > 1]
```


Question:
TypeError: tensor is not a torch image While working through the AI course at Udacity I came across this error during the Transfer Learning section. Here is the code that seems to be causing the trouble:
```
import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
from torchvision import datasets, transforms, models

data_dir = 'filename'

# TODO: Define transforms for the training data and testing data
train_transforms= transforms.Compose([transforms.Resize((224,224)), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), transforms.ToTensor()])
test_transforms= transforms.Compose([transforms.Resize((224,224)), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), transforms.ToTensor()])

# Pass transforms in here, then run the next cell to see how the transforms look
train_data = datasets.ImageFolder(data_dir + '/train', transform=train_transforms)
test_data = datasets.ImageFolder(data_dir + '/test', transform=test_transforms)

trainloader = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)
testloader = torch.utils.data.DataLoader(test_data, batch_size=32)
```

Response:
The problem is with the order of the transforms. The ToTensor transform should come before the Normalize transform, since the latter expects a tensor, but the Resize transform returns an image. Correct code with the faulty lines changed:
```
train_transforms = transforms.Compose([
    transforms.Resize((224,224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])
test_transforms = transforms.Compose([
    transforms.Resize((224,224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])
```


Question: How to index/slice the last dimension of a PyTorch tensor/numpy array of unknown dimensions For example, if I have a 2D tensor X, I can do slicing X[:,1:]; if I have a 3D tensor Y, then I can do similar slicing for the last dimension like Y[:,:,1:]. What is the right way to do the slicing when given a tensor Z of unknown dimension? How about a numpy array? Thanks!

Response:
PyTorch support NumPy-like indexing so you can use Ellipsis(...)
```

```python
z[..., -1:]
#Output
#```
#Example:
#```
```

```python
x                     # (2,2) tensor
#Output
#tensor([[0.5385, 0.9280],
#        [0.8937, 0.0423]])
```

```python
x[..., -1:]
#Output
#tensor([[0.9280],
#        [0.0423]])
#```
#```
```

```python
y                     # (2,2,2) tensor
#Output
#tensor([[[0.5610, 0.8542],
#         [0.2902, 0.2388]],
```

[[0.2440, 0.1063],
         [0.7201, 0.1010]]])

```python
y[..., -1:]
#Output
#tensor([[[0.8542],
#         [0.2388]],
```

[[0.1063],
         [0.1010]]])
```
Ellipsis (...) expands to the number of : objects needed for the selection tuple to index all dimensions. In most cases, this means that length of the expanded selection tuple is x.ndim. There may only be a single ellipsis present.


Question:
Understanding Bilinear Layers When having a bilinear layer in PyTorch I can't wrap my head around how the calculation is done. Here is a small example where I tried to figure out how it works: In:
```
import torch.nn as nn
B = nn.Bilinear(2, 2, 1)
print(B.weight)
```
Out:
```
Parameter containing:
tensor([[[-0.4394, -0.4920],
         [ 0.6137,  0.4174]]], requires_grad=True)
```
I am putting through a zero-vector and a one-vector. In:
```
print(B(torch.ones(2), torch.zeros(2)))
print(B(torch.zeros(2), torch.ones(2)))
```
Out:
```
tensor([0.2175], grad_fn=<ThAddBackward>)
tensor([0.2175], grad_fn=<ThAddBackward>)
```
I tried adding up the weights in various ways but I'm not getting the same result. Thanks in advance!

Response:
The operation done by nn.Bilinear is B(x1, x2) = x1*A*x2 + b (c.f. doc) with: A stored in nn.Bilinear.weight b stored in nn.Bilinear.bias If you take into account the (optional) bias, you should obtain the expected results.
```python
import torch
import torch.nn as nn

def manual_bilinear(x1, x2, A, b):
    return torch.mm(x1, torch.mm(A, x2)) + b

x_ones = torch.ones(2)
x_zeros = torch.zeros(2)

# ---------------------------
# With Bias:

B = nn.Bilinear(2, 2, 1)
A = B.weight
print(B.bias)
# > tensor([-0.6748], requires_grad=True)
b = B.bias

print(B(x_ones, x_zeros))
# > tensor([-0.6748], grad_fn=<ThAddBackward>)
print(manual_bilinear(x_ones.view(1, 2), x_zeros.view(2, 1), A.squeeze(), b))
# > tensor([[-0.6748]], grad_fn=<ThAddBackward>)

print(B(x_ones, x_ones))
# > tensor([-1.7684], grad_fn=<ThAddBackward>)
print(manual_bilinear(x_ones.view(1, 2), x_ones.view(2, 1), A.squeeze(), b))
# > tensor([[-1.7684]], grad_fn=<ThAddBackward>)

# ---------------------------
# Without Bias:

B = nn.Bilinear(2, 2, 1, bias=False)
A = B.weight
print(B.bias)
# None
b = torch.zeros(1)

print(B(x_ones, x_zeros))
# > tensor([0.], grad_fn=<ThAddBackward>)
print(manual_bilinear(x_ones.view(1, 2), x_zeros.view(2, 1), A.squeeze(), b))
# > tensor([0.], grad_fn=<ThAddBackward>)

print(B(x_ones, x_ones))
# > tensor([-0.7897], grad_fn=<ThAddBackward>)
print(manual_bilinear(x_ones.view(1, 2), x_ones.view(2, 1), A.squeeze(), b))
# > tensor([[-0.7897]], grad_fn=<ThAddBackward>)
```


Question: What is the param `last_epoch` on Pytorch Optimizer's Schedulers is for? Regarding Pytorch Optimizer's Schedulers, what actually means the last_epoch argument? It says last_epoch (int) – The index of last epoch. Default: -1. But it doesn't really explains much for those, like me, are just learning about these schedules. I read most of that documentation, if not all, and I could understand what it does and why.

Response:
The last_epoch parameter is used when resuming training and you want to start the scheduler where it left off earlier. Its value is increased every time you call .step() of scheduler. The default value of -1 indicates that the scheduler is started from the beginning. From the docs: Since step() should be invoked after each batch instead of after each epoch, this number represents the total number of batches computed, not the total number of epochs computed. When last_epoch=-1, the schedule is started from the beginning. For example,
```

```python
import torch
```

```python
cc = torch.nn.Conv2d(10,10,3)
```

```python
myoptimizer = torch.optim.Adam(cc.parameters(), lr=0.1)
```

```python
myscheduler = torch.optim.lr_scheduler.StepLR(myoptimizer,step_size=1, gamma=0.1)
```

```python
myscheduler.last_epoch, myscheduler.get_lr()
#Output
#(0, [0.1])
```

```python
myscheduler.step()
```

```python
myscheduler.last_epoch, myscheduler.get_lr()
#Output
#(1, [0.001])
```

```python
myscheduler.step()
```

```python
myscheduler.last_epoch, myscheduler.get_lr()
#Output
#(2, [0.0001])
#```
#Now, if you decide to stop the training in the middle, then resume it, you can provide last_epoch parameter to schedular so that it start from where it was left off, not from the beginning again.
#```
```

```python
mynewscheduler = torch.optim.lr_scheduler.StepLR(myoptimizer,step_size=1, gamma=0.1, last_epoch=myscheduler.last_epoch)
```

```python
mynewscheduler.last_epoch, mynewscheduler.get_lr()
#Output
#(3, [1.0000000000000004e-05])
#```
```


Question:
How to select half precision (BFLOAT16 vs FLOAT16) for your trained model? how will you decide what precision works best for your inference model? Both BF16 and F16 takes two bytes but they use different number of bits for fraction and exponent. Range will be different but I am trying to understand why one chose one over other. Thank you
```
|--------+------+----------+----------|
    | Format | Bits | Exponent | Fraction |
    |--------+------+----------+----------|
    | FP32   |   32 |        8 |       23 |
    | FP16   |   16 |        5 |       10 |
    | BF16   |   16 |        8 |        7 |
    |--------+------+----------+----------|

Range
bfloat16: ~1.18e-38 … ~3.40e38 with 3 significant decimal digits.
float16:  ~5.96e−8 (6.10e−5) … 65504 with 4 significant decimal digits precision.
```

Response: bfloat16 is generally easier to use, because it works as a drop-in replacement for float32. If your code doesn't create nan/inf numbers or turn a non-0 into a 0 with float32, then it shouldn't do it with bfloat16 either, roughly speaking. So, if your hardware supports it, I'd pick that. Check out AMP if you choose float16.


Question:
RuntimeError: "exp" not implemented for 'torch.LongTensor' I am following this tutorial: http://nlp.seas.harvard.edu/2018/04/03/attention.html to implement the Transformer model from the "Attention Is All You Need" paper. However I am getting the following error : RuntimeError: "exp" not implemented for 'torch.LongTensor' This is the line, in the PositionalEnconding class, that is causing the error:
```
div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))
```
When it is being constructed here:
```
pe = PositionalEncoding(20, 0)
```
Any ideas?? I've already tried converting this to perhaps a Tensor Float type, but this has not worked. I've even downloaded the whole notebook with accompanying files and the error seems to persist in the original tutorial. Any ideas what may be causing this error? Thanks!

Response:
I happened to follow this tutorial too. For me I just got the torch.arange to generate float type tensor from
```
position = torch.arange(0, max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))
```
to
```
position = torch.arange(0., max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0., d_model, 2) * -(math.log(10000.0) / d_model))
```
Just a simple fix. But now it works for me. It is possible that the torch exp and sin previously support LongTensor but not anymore (not very sure about it).


Question:
Pytorch: AttributeError: 'function' object has no attribute 'copy' I am trying to load a model state_dict I trained on Google Colab GPU, here is my code to load the model:
```
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

model = models.resnet50()
num_ftrs = model.fc.in_features
model.fc = nn.Linear(num_ftrs, n_classes)
model.load_state_dict(copy.deepcopy(torch.load("./models/model.pth",device)))
model = model.to(device)
model.eval()
```
Here is the error: state_dict = state_dict.copy() AttributeError: 'function' object has no attribute 'copy' Pytorch :
```

```python
import torch
```

```python
print (torch.__version__)
#Output
#1.4.0
```

```python
import torchvision
```

```python
print (torchvision.__version__)
#Output
#0.5.0
#```
#Please help I have searched everywhere to no avail [full error details][1] https://i.sstatic.net/s22DL.png
```

Response:
I am guessing this is what you did by mistake. You saved the function torch.save(model.state_dict, 'model_state.pth') instead of the state_dict() torch.save(model.state_dict(), 'model_state.pth') Otherwise, everything should work as expected. (I tested the following code on Colab) Replace model.state_dict() with model.state_dict to reproduce error
```py
import copy
model = TheModelClass()
torch.save(model.state_dict(), 'model_state.pth')
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model.load_state_dict(copy.deepcopy(torch.load("model_state.pth",device)))
```


Question:
What is the difference between an Embedding Layer with a bias immediately afterwards and a Linear Layer in PyTorch I am reading the "Deep Learning for Coders with fastai & PyTorch" book. I'm still a bit confused as to what the Embedding module does. It seems like a short and simple network, except I can't seem to wrap my head around what Embedding does differently than Linear without a bias. I know it does some faster computational version of a dot product where one of the matrices is a one-hot encoded matrix and the other is the embedding matrix. It does this to in effect select a piece of data? Please point out where I am wrong. Here is one of the simple networks shown in the book.
```
class DotProduct(Module):
    def __init__(self, n_users, n_movies, n_factors):
        self.user_factors = Embedding(n_users, n_factors)
        self.movie_factors = Embedding(n_movies, n_factors)

    def forward(self, x):
        users = self.user_factors(x[:,0])
        movies = self.movie_factors(x[:,1])
        return (users * movies).sum(dim=1)
```

Response:
Embedding [...] what Embedding does differently than Linear without a bias. Essentially everything. torch.nn.Embedding is a lookup table; it works the same as torch.Tensor but with a few twists (like possibility to use sparse embedding or default value at specified index). For example:
```
import torch

embedding = torch.nn.Embedding(3, 4)

print(embedding.weight)

print(embedding(torch.tensor([1])))
```
Would output:
```
Parameter containing:
tensor([[ 0.1420, -0.1886,  0.6524,  0.3079],
        [ 0.2620,  0.4661,  0.7936, -1.6946],
        [ 0.0931,  0.3512,  0.3210, -0.5828]], requires_grad=True)
tensor([[ 0.2620,  0.4661,  0.7936, -1.6946]], grad_fn=<EmbeddingBackward>)
```
So we took the first row of the embedding. It does nothing more than that. Where is it used? Usually when we want to encode some meaning (like word2vec) for each row (e.g. words being close semantically are close in euclidean space) and possibly train them. Linear torch.nn.Linear (without bias) is also a torch.Tensor (weight) but it does operation on it (and the input) which is essentially:
```
output = input.matmul(weight.t())
```
every time you call the layer (see source code and functional definition of this layer). Code snippet The layer in your code snippet does this: creates two lookup tables in __init__ the layer is called with input of shape (batch_size, 2): first column contains indices of user embeddings second column contains indices of movie embeddings these embeddings are multiplied and summed returning (batch_size,) (so it's different from nn.Linear which would return (batch_size, out_features) and perform dot product instead of element-wise multiplication followed by summation like here) This is probably used to train both representations (of users and movies) for some recommender-like system. Other stuff I know it does some faster computational version of a dot product where one of the matrices is a one-hot encoded matrix and the other is the embedding matrix. No, it doesn't. torch.nn.Embedding can be one hot encoded and might also be sparse, but depending on the algorithms (and whether those support sparsity) there might be performance boost or not.


Question:
CNN Pytorch Error : Input type (torch.cuda.ByteTensor) and weight type (torch.cuda.FloatTensor) should be the same I'm receiving the error, Input type (torch.cuda.ByteTensor) and weight type (torch.cuda.FloatTensor) should be the same Following is my code,
```
device    = torch.device('cuda:0')

trainData = torchvision.datasets.FashionMNIST('/content/', train=True, transform=None, target_transform=None, download=True)
testData  = torchvision.datasets.FashionMNIST('/content/', train=False, transform=None, target_transform=None, download=True)

class Net(nn.Module):
  def __init__(self):
    super().__init__()

    '''
    Network Structure:

    input >
    (1)Conv2D > (2)MaxPool2D >
    (3)Conv2D > (4)MaxPool2D >
    (5)Conv2D > (6)MaxPool2D >
    (7)Linear > (8)LinearOut

    '''

    # Creating the convulutional Layers
    self.conv1 = nn.Conv2d(in_channels=CHANNELS, out_channels=32, kernel_size=3)
    self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3)
    self.conv3 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3)

    self.flatten = None
    # Creating a Random dummy sample to get the Flattened Dimensions
    x = torch.randn(CHANNELS, DIM, DIM).view(-1, CHANNELS, DIM, DIM)
    x = self.convs(x)

    # Creating the Linear Layers
    self.fc1   = nn.Linear(self.flatten, 512)
    self.fc2   = nn.Linear(512, CLASSES)

  def convs(self, x):

    # Creating the MaxPooling Layers
    x = F.max_pool2d(F.relu(self.conv1(x)), kernel_size=(2, 2))
    x = F.max_pool2d(F.relu(self.conv2(x)), kernel_size=(2, 2))
    x = F.max_pool2d(F.relu(self.conv3(x)), kernel_size=(2, 2))

    if not self.flatten:
      self.flatten = x[0].shape[0] * x[0].shape[1] * x[0].shape[2]
    return x

  # FORWARD PASS
  def forward(self, x):
    x = self.convs(x)
    x = x.view(-1, self.flatten)
    sm = F.relu(self.fc1(x))
    x = F.softmax(self.fc2(sm), dim=1)
    return x, sm

  x_train, y_train = training_set
  x_train, y_train = x_train.to(device), y_train.to(device)
  optimizer = optim.Adam(net.parameters(), lr=LEARNING_RATE)
  loss_func = nn.MSELoss()
  loss_log  = []

  for epoch in range(EPOCHS):
    for i in tqdm(range(0, len(x_train), BATCH_SIZE)):
        x_batch = x_train[i:i+BATCH_SIZE].view(-1, CHANNELS, DIM, DIM).to(device)
        y_batch = y_train[i:i+BATCH_SIZE].to(device)

        net.zero_grad()
        output, sm = net(x_batch)
        loss = loss_func(output, y_batch.float())
        loss.backward()
        optimizer.step()
    loss_log.append(loss)
    # print(f"Epoch : {epoch} || Loss : {loss}")

  return loss_log

train_set = (trainData.train_data, trainData.train_labels)
test_set  = (testData.test_data, testData.test_labels)

EPOCHS        = 5
LEARNING_RATE = 0.001
BATCH_SIZE    = 32

net = Net().to(device)

loss_log = train(net, train_set, EPOCHS, LEARNING_RATE, BATCH_SIZE)
```
And this is the Error that I'm getting,
```
RuntimeError                              Traceback (most recent call last)
<ipython-input-8-0db1a1b4e37d> in <module>()
      5 net = Net().to(device)
      6
----> 7 loss_log = train(net, train_set, EPOCHS, LEARNING_RATE, BATCH_SIZE)

6 frames
<ipython-input-6-7de4a78e3736> in train(net, training_set, EPOCHS, LEARNING_RATE, BATCH_SIZE)
     13
     14         net.zero_grad()
---> 15         output, sm = net(x_batch)
     16         loss = loss_func(output, y_batch.float())
     17         loss.backward()

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

<ipython-input-5-4fddc427892a> in forward(self, x)
     41   # FORWARD PASS
     42   def forward(self, x):
---> 43     x = self.convs(x)
     44     x = x.view(-1, self.flatten)
     45     sm = F.relu(self.fc1(x))

<ipython-input-5-4fddc427892a> in convs(self, x)
     31
     32     # Creating the MaxPooling Layers
---> 33     x = F.max_pool2d(F.relu(self.conv1(x)), kernel_size=(2, 2))
     34     x = F.max_pool2d(F.relu(self.conv2(x)), kernel_size=(2, 2))
     35     x = F.max_pool2d(F.relu(self.conv3(x)), kernel_size=(2, 2))

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    343
    344     def forward(self, input):
--> 345         return self.conv2d_forward(input, self.weight)
    346
    347 class Conv3d(_ConvNd):

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in conv2d_forward(self, input, weight)
    340                             _pair(0), self.dilation, self.groups)
    341         return F.conv2d(input, weight, self.bias, self.stride,
--> 342                         self.padding, self.dilation, self.groups)
    343
    344     def forward(self, input):

RuntimeError: Input type (torch.cuda.ByteTensor) and weight type (torch.cuda.FloatTensor) should be the same
```
I double-checked that my Neural Net and my Inputs both are in GPU. I'm still getting this error and I don't understand why! Somebody, please help me to get out of this error.

Response: Cast your input x_batch to float. Use x_batch = x_batch.float() before you pass it through your model.


Question:
PyTorch: Extract learned weights correctly I am trying to extract the weights from a linear layer, but they do not appear to change, although error is dropping monotonously (i.e. training is happening). Printing the weights' sum, nothing happens because it stays constant: np.sum(model.fc2.weight.data.numpy()) Here are the code snippets:
```
def train(epochs):
    model.train()
    for epoch in range(1, epochs+1):
        # Train on train set
        print(np.sum(model.fc2.weight.data.numpy()))
        for batch_idx, (data, target) in enumerate(train_loader):
            data, target = Variable(data), Variable(data)
            optimizer.zero_grad()
            output = model(data)
            loss = criterion(output, target)
            loss.backward()
            optimizer.step()
```
and
```
# Define model
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(100, 80, bias=False)
        init.normal(self.fc1.weight, mean=0, std=1)
        self.fc2 = nn.Linear(80, 87)
        self.fc3 = nn.Linear(87, 94)
        self.fc4 = nn.Linear(94, 100)

    def forward(self, x):
        x = self.fc1(x)
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        return x
```
Maybe I am looking on the wrong parameters, although I checked the docs. Thanks for your help!

Response:
Use model.parameters() to get trainable weight for any model or layer. Remember to put it inside list(), or you cannot print it out. The following code snip worked
```

```python
import torch
```

```python
import torch.nn as nn
```

```python
l = nn.Linear(3,5)
```

```python
w = list(l.parameters())
```

```python
w
#Output
#```
```


Question:
Tensorflow: Hierarchical Softmax Implementation I'm currently having text inputs represented by vector, and I want to classify their categories. Because they are multi-level categories, I meant to use Hierarchical Softmax.
```
Example:

 - Computer Science
     - Machine Learning
     - NLP
 - Economics
 - Maths
     - Algebra
     - Geometry
```
I don't know how to implement it in Tensorflow. All examples I've met is using other frameworks. Thanks

Response: Practically if your total number of categories is in the range of hundreds to thousands (less than 50K), you don't need to consider using hierarchical softmax, which is designed to run training faster for classifying into millions of categories (for example, the number of words in a vocabulary). In my experience (with Naive Bayesian and neural networks), utilizing the hierarchical structure at training time does not necessarily improve your classification quality. However, if you are interested to implement Hierarchical Softmax anyway, that's another story.


Question: Pytorch What's the difference between define layer in __init__() and directly use in forward()? What is the difference between the method that define layers in __init__() function, call layer in forward later and the method that directly use layer in forward() function ? Should I define every layer in my compute graph in constructed function(eg. __init__) before I write my compute graph? Could I direct define and use them in forward()?

Response: Everything which contains weights which you want to be trained during the training process should be defined in your __init__ method. You don't need do define activation functions like softmax, ReLU or sigmoid in your __init__, you can just call them in forward. Dropout layers for example also don't need to be defined in __init__, they can just be called in forward too. [However defining them in your __init__ has the advantage that they can be switched off easier during evaluation (by calling eval() on your model). You can see an example of both versions here. Hope this is clear. Just ask if you have any further questions.


Question:
What is the difference between cuda.amp and model.half()? According to https://pytorch.org/blog/accelerating-training-on-nvidia-gpus-with-pytorch-automatic-mixed-precision/ We can use:
```
with torch.cuda.amp.autocast():
      loss = model(data)
```
In order to casts operations to mixed precision. Another thing is that we can use model.half() to convert all the model weights to half precision. What is the difference between these 2 commands ? If I want to take advantage of FP16 (in order to create larger models and shorter training time), what do I need ? Do I need to use model.half() or using torch.cuda.amp (according the link above) ?

Response: If you convert the entire model to fp16, there is a chance that some of the activations functions and batchnorm layers will cause the fp16 weights to underflow, i.e., become zero. So it is always recommended to use autocast which internally converts the weights to fp32 in problematic layers. model.half() in the end will save weight in fp16 where as autocast weights will be still in fp32. Training in fp16 will be faster than autocast but higher chance for instability if you are not careful. While using autocast you also need to scale up the gradient during the back propagation If fp16 requirement is on the inference side, I recommend using autocast and then converting to fp16 using ONNX and tensorrt.


Question: what is the default weight initializer for conv in pytorch? The question How to initialize weights in PyTorch? shows how to initialize the weights in Pytorch. However, what is the default weight initializer for Convand Dense in Pytorch? What distribution does Pytorch use?

Response:
Each pytorch layer implements the method reset_parameters which is called at the end of the layer initialization to initialize the weights. You can find the implementation of the layers here. For the dense layer which in pytorch is called linear for example, weights are initialized uniformly
```
stdv = 1. / math.sqrt(self.weight.size(1))
self.weight.data.uniform_(-stdv, stdv)
```
where self.weight.size(1) is the number of inputs. This is done to keep the variance of the distributions of each layer relatively similar at the beginning of training by normalizing it to one. You can read a more detailed explanation here. For the convolutional layer the initialization is basically the same. You just compute the number of inputs by multiplying the number of channels with the kernel size.


Question:
Why embed dimemsion must be divisible by num of heads in MultiheadAttention? I am learning the Transformer. Here is the pytorch document for MultiheadAttention. In their implementation, I saw there is a constraint:
```
assert self.head_dim * num_heads == self.embed_dim, "embed_dim must be divisible by num_heads"
```
Why require the constraint: embed_dim must be divisible by num_heads? If we go back to the equation Assume: Q, K,V are n x emded_dim matrices; all the weight matrices W is emded_dim x head_dim, Then, the concat [head_i, ..., head_h] will be a n x (num_heads*head_dim) matrix; W^O with size (num_heads*head_dim) x embed_dim [head_i, ..., head_h] * W^O will become a n x embed_dim output I don't know why we require embed_dim must be divisible by num_heads. Let say we have num_heads=10000, the resuts are the same, since the matrix-matrix product will absort this information.

Response:
From what I understood, it is a simplification they have added to keep things simple. Theoretically, we can implement the model like you proposed (similar to the original paper). In pytorch documention, they have briefly mentioned it.
```
Note that `embed_dim` will be split across `num_heads` (i.e. each head will have dimension `embed_dim` // `num_heads`)
```
Also, if you see the Pytorch implementation, you can see it is a bit different (optimised in my point of view) when comparing to the originally proposed model. For example, they use MatMul instead of Linear and Concat layer is ignored. Refer the below which shows the first encoder (with Btach size 32, 10 words, 512 features). P.s: If you need to see the model params (like the above image), this is the code I used.
```
import torch
transformer_model = torch.nn.Transformer(d_model=512, nhead=8, num_encoder_layers=1,num_decoder_layers=1,dim_feedforward=11)  # change params as necessary
tgt = torch.rand((20, 32, 512))
src = torch.rand((11, 32, 512))
torch.onnx.export(transformer_model, (src, tgt), "transformer_model.onnx")
```


Question: RuntimeError: Expected a 'cuda' device type for generator but found 'cpu' I am trying to train PeleeNet pytorch and got the following error train.py line 80 pelee_voc train configuration

Response:
Reading the link provided in @Dwijay 's answer, I found an answer that does not require you to do any source code change. Indeed, it is very dangerous I would say to change PyTorch source code. But the idea of modifying the Generator is the good one. Indeed by default the random number generator generates numbers on CPU, but we want them on GPU. Therefore, one should actually modify the data loader instantiation to fit the use of the default cuda device. This is highlighted in this GitHub comment:
```py
data_loader = data.DataLoader(
    ...,
    generator=torch.Generator(device='cuda'),
)
```
This fix worked for me in PyTorch 1.11 (and worked for this other user in PyTorch 1.10).


Question:
How do I add some Gaussian noise to a tensor in PyTorch? I have a tensor I created using
```
temp = torch.zeros(5, 10, 20, dtype=torch.float64)
    ## some values I set in temp
```
Now I want to add to each temp[i,j,k] a Gaussian noise (sampled from normal distribution with mean 0 and variance 0.1). How do I do it? I would expect there is a function to noise a tensor, but couldn't find anything. I did find this: How to add Poisson noise and Gaussian noise? but it seems to be related to images.

Response:
The function torch.randn produces a tensor with elements drawn from a Gaussian distribution of zero mean and unit variance. Multiply by sqrt(0.1) to have the desired variance.
```
x = torch.zeros(5, 10, 20, dtype=torch.float64)
x = x + (0.1**0.5)*torch.randn(5, 10, 20)
```


Question: How to convert a list or numpy array to a 1d torch tensor? I have a list (or, a numpy array) of float values. I want to create a 1d torch tensor that will contain all those values. I can create the torch tensor and run a loop to store the values. But I want to know is there any way, I can create a torch tensor with initial values from a list or array? Also suggest me if there is any pythonic way to achieve this as I am working in pytorch.

Response:
These are general operations in pytorch and available in the documentation. PyTorch allows easy interfacing with numpy. There is a method called from_numpy and the documentation is available here
```python
import numpy as np
import torch
array = np.arange(1, 11)
tensor = torch.from_numpy(array)
```