Question: Adam optimizer with warmup on PyTorch In the paper Attention is all you need, under section 5.3, the authors suggested to increase the learning rate linearly and then decrease proportionally to the inverse square root of steps. How do we implement this in PyTorch with Adam optimizer? Preferably without additional packages.

Response: PyTorch provides learning-rate-schedulers for implementing various methods of adjusting the learning rate during the training process. Some simple LR-schedulers are are already implemented and can be found here: https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate In your special case you can - just like the other LR-schedulers do - subclass _LRScheduler for implementing a variable schedule based on the number of epochs. For a bare-bones method you only need to implement __init__() and get_lr() methods. Just note that many of these schedulers expect you to call .step() once per epoch. But you can also update it more frequently or even pass a custom argument just like in the cosine-annealing LR-scheduler: https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#CosineAnnealingLR


Question:
"RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 3 3, but got 3-dimensional input of size [3, 224, 224] instead"? I am trying to use a pre-trained model. Here's where the problem occurs Isn't the model supposed to take in a simple colored image? Why is it expecting a 4-dimensional input?
```
RuntimeError                              Traceback (most recent call last)
<ipython-input-51-d7abe3ef1355> in <module>()
     33
     34 # Forward pass the data through the model
---> 35 output = model(data)
     36 init_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability
     37

5 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    336                             _pair(0), self.dilation, self.groups)
    337         return F.conv2d(input, self.weight, self.bias, self.stride,
--> 338                         self.padding, self.dilation, self.groups)
    339
    340

RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 3 3, but got 3-dimensional input of size [3, 224, 224] instead
```
Where
```
inception = models.inception_v3()
model = inception.to(device)
```

Response:
From the Pytorch documentation on convolutional layers, Conv2d layers expect input with the shape
```
(n_samples, channels, height, width) # e.g., (1000, 1, 224, 224)
```
Passing grayscale images in their usual format (224, 224) won't work. To get the right shape, you will need to add a channel dimension. You can do it as follows:
```
x = np.expand_dims(x, 1)      # if numpy array
tensor = tensor.unsqueeze(1)  # if torch tensor
```
The unsqueeze() method adds a dimensions at the specified index. The result would have shape:
```
(1000, 1, 224, 224)
```


Question:
Validation dataset in PyTorch using DataLoaders I want to load the MNIST dataset in PyTorch and Torchvision, dividing it into train, validation, and test parts. So far I have:
```py
def load_dataset():
    train_loader = torch.utils.data.DataLoader(
        torchvision.datasets.MNIST(
            '/data/', train=True, download=True,
            transform=torchvision.transforms.Compose([
                torchvision.transforms.ToTensor()])),
        batch_size=batch_size_train, shuffle=True)

    test_loader = torch.utils.data.DataLoader(
        torchvision.datasets.MNIST(
            '/data/', train=False, download=True,
            transform=torchvision.transforms.Compose([
                torchvision.transforms.ToTensor()])),
        batch_size=batch_size_test, shuffle=True)
```
How can I divide the training dataset into training and validation if it's in the DataLoader? I want to use the last 10000 examples from the training dataset as a validation dataset (I know that I should do a CV for more accurate results, I just want a quick validation here).

Response:
Splitting the training dataset into training and validation in PyTorch turns out to be much harder than it should be. First, split the training set into training and validation subsets (class Subset), which are not datasets (class Dataset):
```
train_subset, val_subset = torch.utils.data.random_split(
        train, [50000, 10000], generator=torch.Generator().manual_seed(1))
```
Then get actual data from those datasets:
```
X_train = train_subset.dataset.data[train_subset.indices]
y_train = train_subset.dataset.targets[train_subset.indices]

X_val = val_subset.dataset.data[val_subset.indices]
y_val = val_subset.dataset.targets[val_subset.indices]
```
Note that this way we don't have Dataset objects, so we can't use DataLoader objects for batch training. If you want to use DataLoaders, they work directly with Subsets:
```
train_loader = DataLoader(dataset=train_subset, shuffle=True, batch_size=BATCH_SIZE)
val_loader = DataLoader(dataset=val_subset, shuffle=False, batch_size=BATCH_SIZE)
```


Question:
How does pytorch's nn.Module register submodule? When I read the source code(python) of torch.nn.Module , I found the attribute self._modules has been used in many functions like self.modules(), self.children(), etc. However, I didn't find any functions updating it. So, where will the self._modules be updated? Furthermore, how does pytorch's nn.Module register submodule?
```
class Module(object):
    def __init__(self):
        self._backend = thnn_backend
        self._parameters = OrderedDict()
        self._buffers = OrderedDict()
        self._backward_hooks = OrderedDict()
        self._forward_hooks = OrderedDict()
        self._forward_pre_hooks = OrderedDict()
        self._modules = OrderedDict()
        self.training = True

    def named_modules(self, memo=None, prefix=''):
        if memo is None:
            memo = set()
        if self not in memo:
            memo.add(self)
            yield prefix, self
            for name, module in self._modules.items():
                if module is None:
                    continue
                submodule_prefix = prefix + ('.' if prefix else '') + name
                for m in module.named_modules(memo, submodule_prefix):
                    yield m
```

Response:
Add some details to Jiren Jin's answer: Layers of a net (inherited from nn.Module) are stored in Module._modules, which is initialized in __construct:
```py
def __init__(self):
    self.__construct()
    # initialize self.training separately from the rest of the internal
    # state, as it is managed differently by nn.Module and ScriptModule
    self.training = True

def __construct(self):
    """
    Initializes internal Module state, shared by both nn.Module and ScriptModule.
    """
    # ...
    self._modules = OrderedDict()
```
self._modules is updated in __setattr__. __setattr__(obj, name, value) is called when obj.name = value is executed. For example, if one defines self.conv1 = nn.Conv2d(128, 256, 3, 1, 1) when initializing a net inherited from nn.Module, the following code from nn.Module.__setattr__ will be executed:
```py
def __setattr__(self, name, value):
    def remove_from(*dicts):
        for d in dicts:
            if name in d:
                del d[name]

    params = self.__dict__.get('_parameters')
    if isinstance(value, Parameter):
        # ...
    elif params is not None and name in params:
        # ...
    else:
        modules = self.__dict__.get('_modules') # equivalent to modules = self._modules
        if isinstance(value, Module):
            if modules is None:
                raise AttributeError(
                    "cannot assign module before Module.__init__() call")
            remove_from(self.__dict__, self._parameters, self._buffers)
            # register the given layer (nn.Conv2d) with its name (conv1)
            # equivalent to self._modules['conv1'] = nn.Conv2d(128, 256, 3, 1, 1)
            modules[name] = value
```
Question from comments: Do you know how this works with the fact that torch lets you supply your own forward method? If one runs a forward pass of a net inherited from nn.Module, the nn.Module.__call__ will be called, in which self.forward is called. However, one has overrided the forward when implementing the net.


Question:
`stack()` vs `cat()` in PyTorch OpenAI's REINFORCE and actor-critic example for reinforcement learning has the following code: REINFORCE:
```py
policy_loss = torch.cat(policy_loss).sum()
```
actor-critic:
```py
loss = torch.stack(policy_losses).sum() + torch.stack(value_losses).sum()
```
One is using torch.cat, the other uses torch.stack, for similar use cases. As far as my understanding goes, the doc doesn't give any clear distinction between them. I would be happy to know the differences between the functions.

Response: stack Concatenates sequence of tensors along a new dimension. cat Concatenates the given sequence of seq tensors in the given dimension. So if A and B are of shape (3, 4): torch.cat([A, B], dim=0) will be of shape (6, 4) torch.stack([A, B], dim=0) will be of shape (2, 3, 4)


Question: How to get the output from a specific layer from a PyTorch model? How to extract the features from a specific layer from a pre-trained PyTorch model (such as ResNet or VGG), without doing a forward pass again?

Response:
New answer Edit: there's a new feature in torchvision v0.11.0 that allows extracting features. For example, if you wanna extract features from the layer layer4.2.relu_2, you can do like:
```py
import torch
from torchvision.models import resnet50
from torchvision.models.feature_extraction import create_feature_extractor

x = torch.rand(1, 3, 224, 224)

model = resnet50()

return_nodes = {
    "layer4.2.relu_2": "layer4"
}
model2 = create_feature_extractor(model, return_nodes=return_nodes)
intermediate_outputs = model2(x)
```
Old answer You can register a forward hook on the specific layer you want. Something like:
```py
def some_specific_layer_hook(module, input_, output):
    pass  # the value is in 'output'

model.some_specific_layer.register_forward_hook(some_specific_layer_hook)

model(some_input)
```
For example, to obtain the res5c output in ResNet, you may want to use a nonlocal variable (or global in Python 2):
```py
res5c_output = None

def res5c_hook(module, input_, output):
    nonlocal res5c_output
    res5c_output = output

resnet.layer4.register_forward_hook(res5c_hook)

resnet(some_input)

# Then, use `res5c_output`.
```


Question:
Different methods for initializing embedding layer weights in Pytorch There seem to be two ways of initializing embedding layers in Pytorch 1.0 using an uniform distribution. For example you have an embedding layer:
```
self.in_embed = nn.Embedding(n_vocab, n_embed)
```
And you want to initialize its weights with an uniform distribution. The first way you can get this done is:
```
self.in_embed.weight.data.uniform_(-1, 1)
```
And another one would be:
```
nn.init.uniform_(self.in_embed.weight, -1.0, 1.0)
```
My question is: what is the difference between the first and second initialization form. Do both methods do the same thing?

Response:
Both are same
```
torch.manual_seed(3)
emb1 = nn.Embedding(5,5)
emb1.weight.data.uniform_(-1, 1)

torch.manual_seed(3)
emb2 = nn.Embedding(5,5)
nn.init.uniform_(emb2.weight, -1.0, 1.0)

assert torch.sum(torch.abs(emb1.weight.data - emb2.weight.data)).numpy() == 0
```
Every tensor has a uniform_ method which initializes it with the values from the uniform distribution. Also, the nn.init module has a method uniform_ which takes in a tensor and inits it with values from uniform distribution. Both are same expect first one is using the member function and the second is using a general utility function.


Question: Pytorch: how to add L1 regularizer to activations? I would like to add the L1 regularizer to the activations output from a ReLU. More generally, how does one add a regularizer only to a particular layer in the network? Related material: This similar post refers to adding L2 regularization, but it appears to add the regularization penalty to all layers of the network. nn.modules.loss.L1Loss() seems relevant, but I do not yet understand how to use this. The legacy module L1Penalty seems relevant also, but why has it been deprecated?

Response:
Here is how you do this: In your Module's forward return final output and layers' output for which you want to apply L1 regularization loss variable will be sum of cross entropy loss of output w.r.t. targets and L1 penalties. Here's an example code
```
import torch
from torch.autograd import Variable
from torch.nn import functional as F

class MLP(torch.nn.Module):
    def __init__(self):
        super(MLP, self).__init__()
        self.linear1 = torch.nn.Linear(128, 32)
        self.linear2 = torch.nn.Linear(32, 16)
        self.linear3 = torch.nn.Linear(16, 2)

    def forward(self, x):
        layer1_out = F.relu(self.linear1(x))
        layer2_out = F.relu(self.linear2(layer1_out))
        out = self.linear3(layer2_out)
        return out, layer1_out, layer2_out

batchsize = 4
lambda1, lambda2 = 0.5, 0.01

model = MLP()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-4)

# usually following code is looped over all batches
# but let's just do a dummy batch for brevity

inputs = Variable(torch.rand(batchsize, 128))
targets = Variable(torch.ones(batchsize).long())

optimizer.zero_grad()
outputs, layer1_out, layer2_out = model(inputs)
cross_entropy_loss = F.cross_entropy(outputs, targets)

all_linear1_params = torch.cat([x.view(-1) for x in model.linear1.parameters()])
all_linear2_params = torch.cat([x.view(-1) for x in model.linear2.parameters()])
l1_regularization = lambda1 * torch.norm(all_linear1_params, 1)
l2_regularization = lambda2 * torch.norm(all_linear2_params, 2)

loss = cross_entropy_loss + l1_regularization + l2_regularization
loss.backward()
optimizer.step()
```


Question: Does batch normalisation work with a small batch size? [closed] Closed. This question is not about programming or software development. It is not currently accepting answers. This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered. Closed 10 months ago. Improve this question I'm using batch normalization with batch size 10 for face detection. Does batch normalization works with such small batch sizes? If not, then what else can i use for normalization?

Response: This question depends mainly on the depth of your neural network. Batch normalization is useful for increasing the training of your data when there are a lot of hidden layers. It decreases the number of epochs required for training the model and helps to regulate the data. By standardizing the inputs to your network, you reduce the risk of chasing a 'moving target', which optimizes the learning process of the model. My advice would be to include batch normalization layers in your code if you have a deep neural network. Reminder, you should probably include some Dropout in your layers as well.


Question:
Correct way to register a parameter for model in Pytorch I tried to define a simple model in Pytorch. The model computes negative log prob for a gaussian distribution:
```
import torch
import torch.nn as nn

class GaussianModel(nn.Module):

    def __init__(self):
        super(GaussianModel, self).__init__()

        self.register_parameter('mean', nn.Parameter(torch.zeros(1),
                                                     requires_grad=True))

        self.pdf = torch.distributions.Normal(self.state_dict()['mean'],
                                              torch.tensor([1.0]))
    def forward(self, x):
        return -self.pdf.log_prob(x)

model = GaussianModel()
```
Then I tried to optimize the mean parameter:
```
optimizer = torch.optim.SGD(model.parameters(), lr=0.002)
for _ in range(5):
  optimizer.zero_grad()
  nll = model(torch.tensor([3.0], requires_grad=True))
  nll.backward()
  optimizer.step()
  print('mean : ', model.state_dict()['mean'],
                 ' - Negative Loglikelihood : ', nll.item())
```
But it seems the gradient is zero and mean does not change:
```
mean :  tensor([0.])  - Negative Loglikelihood :  5.418938636779785
mean :  tensor([0.])  - Negative Loglikelihood :  5.418938636779785
mean :  tensor([0.])  - Negative Loglikelihood :  5.418938636779785
mean :  tensor([0.])  - Negative Loglikelihood :  5.418938636779785
mean :  tensor([0.])  - Negative Loglikelihood :  5.418938636779785
```
Did I register and use the mean parameter correctly? can autograd compute the gradient for torch.distributions.Normal.log_prob or I should implement the backward() for the model?

Response:
You're over complicating registering your parameter. You can just assign a new self.mean attribute to be an nn.Parameter then use it like a tensor for the most part. nn.Module overrides the __setattr__ method which is called every time you assign a new class attribute. One of the things it does is check to see if you assigned an nn.Parameter type, and if so, it adds it to the modules dictionary of registered parameters. Because of this, the easiest way to register your parameter is as follows:
```
import torch
import torch.nn as nn

class GaussianModel(nn.Module):
    def __init__(self):
        super(GaussianModel, self).__init__()
        self.mean = nn.Parameter(torch.zeros(1))
        self.pdf = torch.distributions.Normal(self.mean, torch.tensor([1.0]))

    def forward(self, x):
        return -self.pdf.log_prob(x)
```


Question: Which PyTorch modules are affected by model.eval() and model.train()? The model.eval() method modifies certain modules (layers) which are required to behave differently during training and inference. Some examples are listed in the docs: This has [an] effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc. Is there an exhaustive list of which modules are affected?

Response: In addition to info provided by @iacob: Base class Module Criteria RNNBase RNN LSTM GRU dropout > 0 (default: 0) Transformer layers Transformer TransformerEncoder TransformerDecoder dropout > 0 (Transformer default: 0.1) Lazy variants LazyBatchNorm currently nightly merged PR track_running_stats=True


Question: What is running loss in PyTorch and how is it calculated I had a look at this tutorial in the PyTorch docs for understanding Transfer Learning. There was one line that I failed to understand. After the loss is calculated using loss = criterion(outputs, labels), the running loss is calculated using running_loss += loss.item() * inputs.size(0) and finally, the epoch loss is calculated using running_loss / dataset_sizes[phase]. Isn't loss.item() supposed to be for an entire mini-batch (please correct me if I am wrong). i.e, if the batch_size is 4, loss.item() would give the loss for the entire set of 4 images. If this is true, why is loss.item() being multiplied with inputs.size(0) while calculating running_loss? Isn't this step like an extra multiplication in this case? Any help would be appreciated. Thanks!

Response:
It's because the loss given by CrossEntropy or other loss functions is divided by the number of elements i.e. the reduction parameter is mean by default.
```py
torch.nn.CrossEntropyLoss(weight=None, size_average=None, ignore_index=-100, reduce=None, reduction='mean')
```
Hence, loss.item() contains the loss of entire mini-batch, but divided by the batch size. That's why loss.item() is multiplied with batch size, given by inputs.size(0), while calculating running_loss.


Question:
early stopping in PyTorch I tried to implement an early stopping function to avoid my neural network model overfit. I'm pretty sure that the logic is fine, but for some reason, it doesn't work. I want that when the validation loss is greater than the training loss over some epochs, the early stopping function returns True. But it returns False all the time, even though the validation loss becomes a lot greater than the training loss. Could you see where is the problem, please? early stopping function
```
def early_stopping(train_loss, validation_loss, min_delta, tolerance):

    counter = 0
    if (validation_loss - train_loss) > min_delta:
        counter +=1
        if counter >= tolerance:
          return True
```
calling the function during the training
```
for i in range(epochs):

    print(f"Epoch {i+1}")
    epoch_train_loss, pred = train_one_epoch(model, train_dataloader, loss_func, optimiser, device)
    train_loss.append(epoch_train_loss)

    # validation

    with torch.no_grad():
       epoch_validate_loss = validate_one_epoch(model, validate_dataloader, loss_func, device)
       validation_loss.append(epoch_validate_loss)

    # early stopping
    if early_stopping(epoch_train_loss, epoch_validate_loss, min_delta=10, tolerance = 20):
      print("We are at epoch:", i)
      break
```
EDIT: The train and validation loss: EDIT2:
```
def train_validate (model, train_dataloader, validate_dataloader, loss_func, optimiser, device, epochs):
    preds = []
    train_loss =  []
    validation_loss = []
    min_delta = 5

    for e in range(epochs):

        print(f"Epoch {e+1}")
        epoch_train_loss, pred = train_one_epoch(model, train_dataloader, loss_func, optimiser, device)
        train_loss.append(epoch_train_loss)

        # validation
        with torch.no_grad():
           epoch_validate_loss = validate_one_epoch(model, validate_dataloader, loss_func, device)
           validation_loss.append(epoch_validate_loss)

        # early stopping
        early_stopping = EarlyStopping(tolerance=2, min_delta=5)
        early_stopping(epoch_train_loss, epoch_validate_loss)
        if early_stopping.early_stop:
            print("We are at epoch:", e)
            break

    return train_loss, validation_loss
```

Response:
Although @KarelZe's response solves your problem sufficiently and elegantly, I want to provide an alternative early stopping criterion that is arguably better. Your early stopping criterion is based on how much (and for how long) the validation loss diverges from the training loss. This will break when the validation loss is indeed decreasing but is generally not close enough to the training loss. The goal of training a model is to encourage the reduction of validation loss and not the reduction in the gap between training loss and validation loss. Hence, I would argue that a better early stopping criterion would be watch for the trend in validation loss alone, i.e., if the training is not resulting in lowering of the validation loss then terminate it. Here's an example implementation:
```
class EarlyStopper:
    def __init__(self, patience=1, min_delta=0):
        self.patience = patience
        self.min_delta = min_delta
        self.counter = 0
        self.min_validation_loss = float('inf')

    def early_stop(self, validation_loss):
        if validation_loss < self.min_validation_loss:
            self.min_validation_loss = validation_loss
            self.counter = 0
        elif validation_loss > (self.min_validation_loss + self.min_delta):
            self.counter += 1
            if self.counter >= self.patience:
                return True
        return False
```
Here's how you'd use it:
```
early_stopper = EarlyStopper(patience=3, min_delta=10)
for epoch in np.arange(n_epochs):
    train_loss = train_one_epoch(model, train_loader)
    validation_loss = validate_one_epoch(model, validation_loader)
    if early_stopper.early_stop(validation_loss):
        break
```


Question:
Tracing back deprecated warning in pytorch I am training yolov3 on my data using this code here : https://github.com/cfotache/pytorch_custom_yolo_training/ But I am getting this annoying deprecation warnings
```
Warning: indexing with dtype torch.uint8 is now deprecated, please use a dtype torch.bool instead. (expandTensors at /pytorch/aten/src/ATen/native/IndexingUtils.h:20)
```
I tried using python3 -W ignore train.py I tried adding :
```
import warnings
warnings.filterwarnings('ignore')
```
but the warning is still persistent. I found this piece of code on here on stackoverflow that prints that stack on warnings ,
```
import traceback
import warnings
import sys

def warn_with_traceback(message, category, filename, lineno, file=None, line=None):

    log = file if hasattr(file,'write') else sys.stderr
    traceback.print_stack(file=log)
    log.write(warnings.formatwarning(message, category, filename, lineno, line))

warnings.showwarning = warn_with_traceback
```
and here's what I get :
```
File "/content/pytorch_custom_yolo_training/train.py", line 102, in <module>
   loss = model(imgs, targets)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/pytorch_custom_yolo_training/models.py", line 267, in forward
    x, *losses = module[0](x, targets)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/pytorch_custom_yolo_training/models.py", line 203, in forward
    loss_x = self.mse_loss(x[mask], tx[mask])
  File "/usr/lib/python3.6/warnings.py", line 99, in _showwarnmsg
    msg.file, msg.line)
  File "/content/pytorch_custom_yolo_training/train.py", line 29, in warn_with_traceback
    traceback.print_stack(file=log)
  /pytorch/aten/src/ATen/native/IndexingUtils.h:20: UserWarning: indexing with dtype torch.uint8 is now deprecated, please use a dtype torch.bool instead.
```
Going to the files and functions mentioned in the stack , I don't find any uint8 . What can I to solve the problem or even stop getting these warnings ?

Response: Found the problem. line : loss_x = self.mse_loss(x[mask], tx[mask]) the mask variable was a ByteTensor which is deprecated . Just replaced it with a BoolTensor


Question:
Finding mean and standard deviation across image channels PyTorch Say I have a batch of images in the form of tensors with dimensions (B x C x W x H) where B is the batch size, C is the number of channels in the image, and W and H are the width and height of the image respectively. I'm looking to use the transforms.Normalize() function to normalize my images with respect to the mean and standard deviation of the dataset across the C image channels, meaning that I want a resulting tensor in the form 1 x C. Is there a straightforward way to do this? I tried torch.view(C, -1).mean(1) and torch.view(C, -1).std(1) but I get the error:
```
view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
```
Edit After looking into how view() works in PyTorch, I know realize why my approach doesn't work; however, I still can't figure out how to get the per-channel mean and standard deviation.

Response:
Note that variances add, not standard deviations. See detailed explanation here: https://apcentral.collegeboard.org/courses/ap-statistics/classroom-resources/why-variances-add-and-why-it-matters Here is the modified code:
```
nimages = 0
mean = 0.0
var = 0.0
for i_batch, batch_target in enumerate(trainloader):
    batch = batch_target[0]
    # Rearrange batch to be the shape of [B, C, W * H]
    batch = batch.view(batch.size(0), batch.size(1), -1)
    # Update total number of images
    nimages += batch.size(0)
    # Compute mean and std here
    mean += batch.mean(2).sum(0)
    var += batch.var(2).sum(0)

mean /= nimages
var /= nimages
std = torch.sqrt(var)

print(mean)
print(std)
```


Question:
Why aren't torch.nn.Parameter listed when net is printed? I recently had to construct a module that required a tensor to be included. While back propagation worked perfectly using torch.nn.Parameter, it did not show up when printing the net object. Why isn't this parameter included in contrast to other modules like layer? (Shouldn't it behave just like layer?)
```
import torch
import torch.nn as nn

class MyNet(torch.nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.layer = nn.Linear(10, 10)
        self.parameter = torch.nn.Parameter(torch.zeros(10,10, requires_grad=True))

net = MyNet()
print(net)
```
Output:
```
MyNet(
  (layer): Linear(in_features=10, out_features=10, bias=True)
)
```

Response:
When you call print(net), the __repr__ method is called. __repr__ gives the “official” string representation of an object. In PyTorch's nn.Module (base class of your MyNet model), the __repr__ is implemented like this:
```
def __repr__(self):
        # We treat the extra repr like the sub-module, one item per line
        extra_lines = []
        extra_repr = self.extra_repr()
        # empty string will be split into list ['']
        if extra_repr:
            extra_lines = extra_repr.split('\n')
        child_lines = []
        for key, module in self._modules.items():
            mod_str = repr(module)
            mod_str = _addindent(mod_str, 2)
            child_lines.append('(' + key + '): ' + mod_str)
        lines = extra_lines + child_lines

        main_str = self._get_name() + '('
        if lines:
            # simple one-liner info, which most builtin Modules will use
            if len(extra_lines) == 1 and not child_lines:
                main_str += extra_lines[0]
            else:
                main_str += '\n  ' + '\n  '.join(lines) + '\n'

        main_str += ')'
        return main_str
```
Note that the above method returns main_str which contains call to only _modules and extra_repr, thus it prints only modules by default. PyTorch also provides extra_repr() method which you can implement yourself for extra representation of the module. To print customized extra information, you should reimplement this method in your own modules. Both single-line and multi-line strings are acceptable.


Question:
Print exact value of PyTorch tensor (floating point precision) I'm trying to print torch.FloatTensor like:
```
a = torch.FloatTensor(3,3)
print(a)
```
This way I can get a value like:
```
0.0000e+00  0.0000e+00  3.2286e-41
1.2412e-40  1.2313e+00  1.6751e-37
2.6801e-36  3.5873e-41  9.4463e+21
```
But I want to get more accurate value, like 10 decimal point:
```
0.1234567891+01
```
With other python numerical objects, I could get it with:
```
print('{:.10f}'.format(a))
```
but in the case of a tensor, I get this error:
```
TypeError: unsupported format string passed to torch.FloatTensor.__format__
```
How can I print more precise values of tensors?

Response:
You can set the precision options:
```
torch.set_printoptions(precision=10)
```
There are more formatting options on the documentation page, it is very similar to numpy's.


Question:
What is volatile variable in Pytorch What is volatile attribute of a Variable in Pytorch? Here's a sample code for defining a variable in PyTorch.
```
datatensor = Variable(data, volatile=True)
```

Response: Basically, set the input to a network to volatile if you are doing inference only and won't be running backpropagation in order to conserve memory. From the docs: Volatile is recommended for purely inference mode, when you’re sure you won’t be even calling .backward(). It’s more efficient than any other autograd setting - it will use the absolute minimal amount of memory to evaluate the model. volatile also determines that requires_grad is False. Edit: The volatile keyword has been deprecated as of pytorch version 0.4.0


Question: How does Pytorch's "Fold" and "Unfold" work? I've gone through the official doc. I'm having a hard time understanding what this function is used for and how it works. Can someone explain this in layman's terms?

Response:
unfold imagines a tensor as a longer tensor with repeated columns/rows of values 'folded' on top of each other, which is then "unfolded": size determines how large the folds are step determines how often it is folded E.g. for a 2x5 tensor, unfolding it with step=1, and patch size=2 across dim=1:
```py
x = torch.tensor([[1,2,3,4,5],
                  [6,7,8,9,10]])
```
```py

```python
x.unfold(1,2,1)
#Output
#tensor([[[ 1,  2], [ 2,  3], [ 3,  4], [ 4,  5]],
#        [[ 6,  7], [ 7,  8], [ 8,  9], [ 9, 10]]])
#```
#fold is roughly the opposite of this operation, but "overlapping" values are summed in the output.
```


Question:
Get probability of multi-token word in MASK position It is relatively easy to get a token's probability according to a language model, as the snippet below shows. You can get the output of a model, restrict yourself to the output of the masked token, and then find the probability of your requested token in the output vector. However, this only works with single-token words, e.g. words that are themselves in the tokenizer's vocabulary. When a word does not exist in the vocabulary, the tokenizer will chunk it up into pieces that it does know (see the bottom of the example). But since the input sentence consists of only one masked position, and the requested token has more tokens than that, how can we get its probability? Ultimately I am looking for a solution that works regardless of the number of subword units a word has. In the code below I have added many comments explaining what is going on, as well as printing out the given output of print statements. You'll see that predicting tokens such as 'love' and 'hate' is straightforward because they are in the tokenizer's vocabulary. 'reprimand' is not, though, so it cannot be predicted in a single masked position - it consists of three subword units. So how can we predict 'reprimand' in the masked position?
```py
from transformers import BertTokenizer, BertForMaskedLM
import torch

# init model and tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
model.eval()
# init softmax to get probabilities later on
sm = torch.nn.Softmax(dim=0)
torch.set_grad_enabled(False)

# set sentence with MASK token, convert to token_ids
sentence = f"I {tokenizer.mask_token} you"
token_ids = tokenizer.encode(sentence, return_tensors='pt')
print(token_ids)
# tensor([[ 101, 1045,  103, 2017,  102]])
# get the position of the masked token
masked_position = (token_ids.squeeze() == tokenizer.mask_token_id).nonzero().item()

# forward
output = model(token_ids)
last_hidden_state = output[0].squeeze(0)
# only get output for masked token
# output is the size of the vocabulary
mask_hidden_state = last_hidden_state[masked_position]
# convert to probabilities (softmax)
# giving a probability for each item in the vocabulary
probs = sm(mask_hidden_state)

# get probability of token 'hate'
hate_id = tokenizer.convert_tokens_to_ids('hate')
print('hate probability', probs[hate_id].item())
# hate probability 0.008057191967964172

# get probability of token 'love'
love_id = tokenizer.convert_tokens_to_ids('love')
print('love probability', probs[love_id].item())
# love probability 0.6704086065292358

# get probability of token 'reprimand' (?)
reprimand_id = tokenizer.convert_tokens_to_ids('reprimand')
# reprimand is not in the vocabulary, so it needs to be split into subword units
print(tokenizer.convert_ids_to_tokens(reprimand_id))
# [UNK]

reprimand_id = tokenizer.encode('reprimand', add_special_tokens=False)
print(tokenizer.convert_ids_to_tokens(reprimand_id))
# ['rep', '##rim', '##and']
# but how do we now get the probability of a multi-token word in a single-token position?
```

Response: Since the split word is not present in the dictionary, BERT is simply unaware of its probability, so there is no use of masking it before tokenization. And you can't get its probability by exploiting the rule of chain, see response by J.Devlin. To illustrate it, let's take a more generic example. Try to estimate the probability of some bigram in position i. While you can estimate the probability of each word given the sentence and their positions P(w_i|w_0, w_1... w_i-1, w_i+1, ..., w_N), P(w_i+1|w_0, w_1... w_i, wi+2, ..., w_N), there is no way to get the probability of the bigram P(w_i,w_i+1|w_0, w_1... w_i-1, wi+2, ..., w_N) because BERT does not store such information. Having said all that, you can get a very rough estimate of the probability of your OOV word by multiplying probabilities of seeing it's parts. So you will get P("reprimand"|...) ~= P("rep"|...)*P("##rim"|...)*P("##and"|...) Since your subwords are not regular words, but a special kind of words, this is not all wrong, because the dependency between them is implicit.


Question:
What does next() and iter() do in PyTorch's DataLoader() I have the following code:
```
import torch
import numpy as np
import pandas as pd
from torch.utils.data import TensorDataset, DataLoader

# Load dataset
df = pd.read_csv(r'../iris.csv')

# Extract features and target
data = df.drop('target',axis=1).values
labels = df['target'].values

# Create tensor dataset
iris = TensorDataset(torch.FloatTensor(data),torch.LongTensor(labels))

# Create random batches
iris_loader = DataLoader(iris, batch_size=105, shuffle=True)

next(iter(iris_loader))
```
What does next() and iter() do in the above code? I have went through PyTorch's documentation and still can't quite understand what is next() and iter() doing here. Can anyone help in explaining this? Many thanks in advance.

Response: These are built-in functions of python, they are used for working with iterables. Basically iter() calls the __iter__() method on the iris_loader which returns an iterator. next() then calls the __next__() method on that iterator to get the first iteration. Running next() again will get the second item of the iterator, etc. This logic often happens 'behind the scenes', for example when running a for loop. It calls the __iter__() method on the iterable, and then calls __next__() on the returned iterator until it reaches the end of the iterator. It then raises a stopIteration and the loop stops. Please see the documentation for further details and some nuances: https://docs.python.org/3/library/functions.html#iter


Question:
PyTorch slice matrix with vector Say I have one matrix and one vector as follows:
```py
import torch
x = torch.tensor([[1, 2, 3],
                  [4, 5, 6],
                  [7, 8, 9]])

y = torch.tensor([0, 2, 1])
```
is there a way to slice it x[y] so the result is:
```py
res = [1, 6, 8]
```
So basically I take the first element of y and take the element in x that corresponds to the first row and the elements' column.

Response:
You can specify the corresponding row index as:
```py
import torch
x = torch.tensor([[1, 2, 3],
                  [4, 5, 6],
                  [7, 8, 9]])

y = torch.tensor([0, 2, 1])

x[range(x.shape[0]), y]
tensor([1, 6, 8])
```


Question: Pytorch: How to compute IoU (Jaccard Index) for semantic segmentation Can someone provide a toy example of how to compute IoU (intersection over union) for semantic segmentation in pytorch?

Response:
As of 2021, there's no need to implement your own IoU, as torchmetrics comes equipped with it - here's the link. It is named torchmetrics.JaccardIndex (previously torchmetrics.IoU) and calculates what you want. It works with PyTorch and PyTorch Lightning, also with distributed training. From the documentation: torchmetrics.JaccardIndex(num_classes, ignore_index=None, absent_score=0.0, threshold=0.5, multilabel=False, reduction='elementwise_mean', compute_on_step=None, **kwargs) Computes Intersection over union, or Jaccard index calculation: J(A,B) = \frac{|A\cap B|}{|A\cup B|} Where: A and B are both tensors of the same size, containing integer class values. They may be subject to conversion from input data (see description below). Note that it is different from box IoU. Works with binary, multiclass and multi-label data. Accepts probabilities from a model output or integer class values in prediction. Works with multi-dimensional preds and target. Forward accepts preds (float or long tensor): (N, ...) or (N, C, ...) where C is the number of classes target (long tensor): (N, ...) If preds and target are the same shape and preds is a float tensor, we use the self.threshold argument to convert into integer labels. This is the case for binary and multi-label probabilities. If preds has an extra dimension as in the case of multi-class scores we perform an argmax on dim=1. Official example:
```py
from torchmetrics import JaccardIndex
target = torch.randint(0, 2, (10, 25, 25))
pred = torch.tensor(target)
pred[2:5, 7:13, 9:15] = 1 - pred[2:5, 7:13, 9:15]
jaccard = JaccardIndex(task='multiclass', num_classes=2)
jaccard(pred, target)
```
Returns tensor(0.9660).


Question:
Where is the source code of pytorch conv2d? Where do I find the source code of the pytorch function conv2d? It should be in torch.nn.functional but I only find _add_docstr lines, if i search for conv2d. I looked here: https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py Update: It's is not my typing, I do mean the function. Conv2d class uses conv2d function from the nn.functional Here: https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/conv.py On line 338:
```
return F.conv2d(F.pad(input, expanded_padding, mode='circular')
```
F is how they import functional So then I went there, but I don't find the code.

Response: The functional code is implemented in C++. As of version 1.13.1 the entry point into the C++ code for conv2d is at aten/src/ATen/native/Convolution.cpp:804. If you are interested more generally in how functions are registered to the API then you can take a look at aten/src/ATen/native/README.md. A deeper dive will benefit from understanding some of the design decisions in PyTorch. For example, the dispatcher mechanism (see here). More general information can be found in the PyTorch developer's wiki, though keep in mind this wiki is primarily a tool for contributors and is not as polished as the Python API documentation. IMO a good starting point is the Core Frontend Onboarding page which gives links to most everything needed to get your head around the PyTorch source code.


Question:
How Pytorch Tensor get the index of specific value With python lists, we can do:
```
a = [1, 2, 3]
assert a.index(2) == 1
```
How can a pytorch tensor find the .index() directly?

Response:
I think there is no direct translation from list.index() to a pytorch function. However, you can achieve similar results using tensor==number and then the nonzero() function. For example:
```
t = torch.Tensor([1, 2, 3])
print ((t == 2).nonzero(as_tuple=True)[0])
```
This piece of code returns 1 [torch.LongTensor of size 1x1]


Question: Writing tensor to a file in a visually readable manner In pytorch, I want to write a tensor to a file and visually read the file contents. For example, consider T = torch.tensor([3,4,5,6]). I want to write the tensor T to a file, say file_T.txt, and want to visually read the contents of the file_T.txt, which will be 3,4,5 and 6. How can I achieve this?

Response:
You can use numpy:
```
import numpy as np
np.savetxt('my_file.txt', torch.Tensor([3,4,5,6]).numpy())
```


Question: Parallelization strategies for deep learning What strategies and forms of parallelization are feasible and available for training and serving a neural network?: inside a machine across cores (e.g. GPU / TPU / CPU) across machines on a network or a rack I'm also looking for evidence for how they may also be used in e.g. TensorFlow, PyTorch or MXNet. Training To my knowledge, when training large neural networks on large datasets, one could at least have: Different cores or machines operate on different parts of the graph ("graph splitting"). E.g. backpropagation through the graph itself can be parallelized e.g. by having different layers hosted on different machines since (I think?) the autodiff graph is always a DAG. Different cores or machines operate on different samples of data ("data splitting"). In SGD, the computation of gradients across batches or samples can also be parallelized (e.g. the gradients can be combined after computing them independently on different batches). I believe this is also called gradient accumulation (?). When is each strategy better for what type of problem or neural network? Which modes are supported by modern libraries? and can one combine all four (2x2) strategies? On top of that, I have read about: Asynchronous training Synchronous training but I don't know what exactly that refers to, e.g. is it the computation of gradients on different data batches or the computation of gradients on different subgraphs? Or perhaps it refers to something else altogether? Serving If the network is huge, prediction / inference may also be slow, and the model may not fit on a single machine in memory at serving time. Are there any known multi-core and multi-node prediction solutions that work that can handle such models?

Response:
Training In general, there are two strategies of parallelizing model training: data parallelism and model parallelism. 1. Data parallelism This strategy splits training data into N partitions, each of which will be trained on different “devices” (different CPU cores, GPUs, or even machines). In contrast to training without data parallelism which produces one gradient per minibatch, we now have N gradients for each minibatch step. The next question is how we should combine these N gradients. One way to do it is by averaging all the N gradients and then updating the model parameters once based on the average. This technique is called synchronous distributed SGD. By doing the average, we have a more accurate gradient, but with a cost of waiting all the devices to finish computing its own local gradient. Another way is by not combining the gradients — each gradient will instead be used to update the model parameters independently. So, there will be N parameter updates for each minibatch step, in contrast to only one for the previous technique. This technique is called asynchronous distributed SGD. Because it doesn't have to wait other devices to finish, the async approach will take less time to complete a minibatch step than the sync approach will do. However, the async approach will produce a more noisy gradient, so it might need to complete more minibatch steps to catch up with the performance (in terms of loss) of the sync approach. There are many papers proposing some improvements and optimizations on either approach, but the main idea is generally the same as described above. In the literature there's been some disagreement on which technique is better in practice. At the end most people now settle on the synchronous approach. Data Parallelism in PyTorch To do synchronous SGD, we can wrap our model with torch.nn.parallel.DistributedDataParallel:
```
from torch.nn.parallel import DistributedDataParallel as DDP

# `model` is the model we previously initialized
model = ...

# `rank` is a device number starting from 0
model = model.to(rank)
ddp_model = DDP(model, device_ids=[rank])
```
Then we can train it similarly. For more details, you can refer to the official tutorial. For doing asynchronous SGD in PyTorch, we need to implement it more manually since there is no wrapper similar to DistributedDataParallel for it. Data Parallelism in TensorFlow/Keras For synchronous SGD, we can use tf.distribute.MirroredStrategy to wrap the model initalization:
```
import tensorflow as tf

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    model = Model(...)
    model.compile(...)
```
Then we can train it as usual. For more details, you can refer to the official guides on Keras website and TensorFlow website. For asynchronous SGD, we can use tf.distribute.experimental.ParameterServerStrategy similarly. 2. Model Parallelism This strategy splits the model into N parts, each of which will be computed on different devices. A common way to split the model is based on layers: different sets of layers are placed on different devices. But we can also split it more intricately depending on the model architecture. Model Parallelism in TensorFlow and PyTorch To implement model parallelism in either TensorFlow or PyTorch, the idea is the same: to move some model parameters into a different device. In PyTorch we can use torch.nn.Module.to method to move a module into a different device. For example, suppose we want to create two linear layers each of which is placed on a different GPU:
```
import torch.nn as nn

linear1 = nn.Linear(16, 8).to('cuda:0')
linear2 = nn.Linear(8, 4).to('cuda:1')
```
In TensorFlow we can use tf.device to place an operation into a specific device. To implement the PyTorch example above in TensorFlow:
```
import tensorflow as tf
from tensorflow.keras import layers

with tf.device('/GPU:0'):
    linear1 = layers.Dense(8, input_dim=16)
with tf.device('/GPU:1'):
    linear2 = layers.Dense(4, input_dim=8)
```
For more details you can refer to the official PyTorch tutorial; or if you use TensorFlow you can even use a more high-level library like mesh. 3. Hybrid: Data and Model Parallelism Recall that data parallelism only splits the training data, whereas model parallelism only splits the model structures. If we have a model so large that even after using either parallelism strategy it still doesn't fit in the memory, we can always do both. In practice most people prefer data parallelism to model parallelism since the former is more decoupled (in fact, independent) from the model architecture than the latter. That is, by using data parallelism they can change the model architecture as they like, without worrying which part of the model should be parallelized. Model Inference / Serving Parallelizing model serving is easier than parallelizing model training since the model parameters are already fixed and each request can be processed independently. Similar to scaling a regular Python web service, we can scale model serving by spawning more processes (to workaround Python's GIL) in a single machine, or even spawning more machine instances. When we use a GPU to serve the model, though, we need to do more work to scale it. Because of how concurrency is handled differently by a GPU compared to a CPU, in order to maximize the performance, we need to do inference request batching. The idea is when a request comes, instead of immediately processing it, we wait some timeout duration for other requests to come. When the timeout is up, even if the number of requests is only one, we batch them all to be processed on the GPU. In order to minimize the average request latency, we need to find the optimal timeout duration. To find it we need to observe that there is a trade-off between minimizing the timeout duration and maximizing the number of batch size. If the timeout is too low, the batch size will be small, so the GPU will be underutilized. But if the timeout is too high, the requests that come early will wait too long before they get processed. So, the optimal timeout duration depends on the model complexity (hence, the inference duration) and the average requests per second to receive. Implementing a scheduler to do request batching is not a trivial task, so instead of doing it manually, we'd better use TensorFlow Serving or PyTorch Serve which already supports it. To learn more about parallel and distributed learning, you can read this review paper.


Question:
What does the parameter retain_graph mean in the Variable's backward() method? I'm going through the neural transfer pytorch tutorial and am confused about the use of retain_variable(deprecated, now referred to as retain_graph). The code example show:
```
class ContentLoss(nn.Module):

    def __init__(self, target, weight):
        super(ContentLoss, self).__init__()
        self.target = target.detach() * weight
        self.weight = weight
        self.criterion = nn.MSELoss()

    def forward(self, input):
        self.loss = self.criterion(input * self.weight, self.target)
        self.output = input
        return self.output

    def backward(self, retain_variables=True):
        #Why is retain_variables True??
        self.loss.backward(retain_variables=retain_variables)
        return self.loss
```
From the documentation retain_graph (bool, optional) – If False, the graph used to compute the grad will be freed. Note that in nearly all cases setting this option to True is not needed and often can be worked around in a much more efficient way. Defaults to the value of create_graph. So by setting retain_graph= True, we're not freeing the memory allocated for the graph on the backward pass. What is the advantage of keeping this memory around, why do we need it?

Response:
@cleros is pretty on the point about the use of retain_graph=True. In essence, it will retain any necessary information to calculate a certain variable, so that we can do backward pass on it. An illustrative example Suppose that we have a computation graph shown above. The variable d and e is the output, and a is the input. For example,
```py
import torch
from torch.autograd import Variable
a = Variable(torch.rand(1, 4), requires_grad=True)
b = a**2
c = b*2
d = c.mean()
e = c.sum()
```
when we do d.backward(), that is fine. After this computation, the parts of the graph that calculate d will be freed by default to save memory. So if we do e.backward(), the error message will pop up. In order to do e.backward(), we have to set the parameter retain_graph to True in d.backward(), i.e.,
```py
d.backward(retain_graph=True)
```
As long as you use retain_graph=True in your backward method, you can do backward any time you want:
```py
d.backward(retain_graph=True) # fine
e.backward(retain_graph=True) # fine
d.backward() # also fine
e.backward() # error will occur!
```
More useful discussion can be found here. A real use case Right now, a real use case is multi-task learning where you have multiple losses that maybe be at different layers. Suppose that you have 2 losses: loss1 and loss2 and they reside in different layers. In order to backprop the gradient of loss1 and loss2 w.r.t to the learnable weight of your network independently. You have to use retain_graph=True in backward() method in the first back-propagated loss.
```py
# suppose you first back-propagate loss1, then loss2 (you can also do the reverse)
loss1.backward(retain_graph=True)
loss2.backward() # now the graph is freed, and next process of batch gradient descent is ready

optimizer.step() # update the network parameters
```


Question: How to find IoU from segmentation masks? I am doing an image segmentation task and I am using a dataset that only has ground truths but no bounding boxes or polygons. I have 2 classes( ignoring 0 for background) and the outputs and ground truth labels are in an array like Predicted--/---Labels 0|0|0|1|2 0|0|0|1|2 0|2|1|0|0 0|2|1|0|0 0|0|1|1|1 0|0|1|1|1 0|0|0|0|1 0|0|0|0|1 How do I calculate IoU from these ? PS: I am using python3 with pytorch api

Response: So I just found out that jaccard_similarity_score is regarded as IoU. So the solution is very simple, from sklearn.metrics import jaccard_similarity_score jac = jaccard_similarity_score(predictions, label, Normalize = True/False) Source link: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_score.html#sklearn.metrics.jaccard_score


Question:
How to save and load random number generator state in Pytorch? I am training a DL model in Pytorch, and want to train my model in a deterministic way. As written in this official guide, I set random seeds like this:
```py
np.random.seed(0)
torch.manual_seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
```
Now, my training is long and i want to save, then later load everything, including the RNGs. I use torch.save and torch.load_state_dict for the model and the optimizer. How can the random number generators be saved & loaded?

Response: You can use torch.get_rng_state and torch.set_rng_state When calling torch.get_rng_state you will get your random number generator state as a torch.ByteTensor. You can then save this tensor somewhere in a file and later you can load and use torch.set_rng_state to set the random number generator state. When using numpy you can of course do the same there using: numpy.random.get_state and numpy.random.set_state


Question:
How to load a list of numpy arrays to pytorch dataset loader? I have a huge list of numpy arrays, where each array represents an image and I want to load it using torch.utils.data.Dataloader object. But the documentation of torch.utils.data.Dataloader mentions that it loads data directly from a folder. How do I modify it for my cause? I am new to pytorch and any help would be greatly appreciated. my numpy array for a single image looks something like this. The image is RBG image.
```
[[[ 70  82  94]
  [ 67  81  93]
  [ 66  82  94]
  ...,
  [182 182 188]
  [183 183 189]
  [188 186 192]]

 [[ 66  80  92]
  [ 62  78  91]
  [ 64  79  95]
  ...,
  [176 176 182]
  [178 178 184]
  [180 180 186]]

 [[ 62  82  93]
  [ 62  81  96]
  [ 65  80  99]
  ...,
  [169 172 177]
  [173 173 179]
  [172 172 178]]

 ...,
```

Response:
I think what DataLoader actually requires is an input that subclasses Dataset. You can either write your own dataset class that subclasses Datasetor use TensorDataset as I have done below:
```
import torch
import numpy as np
from torch.utils.data import TensorDataset, DataLoader

my_x = [np.array([[1.0,2],[3,4]]),np.array([[5.,6],[7,8]])] # a list of numpy arrays
my_y = [np.array([4.]), np.array([2.])] # another list of numpy arrays (targets)

tensor_x = torch.Tensor(my_x) # transform to torch tensor
tensor_y = torch.Tensor(my_y)

my_dataset = TensorDataset(tensor_x,tensor_y) # create your datset
my_dataloader = DataLoader(my_dataset) # create your dataloader
```
Works for me.


Question:
RuntimeError: module must have its parameters and buffers on device cuda:1 (device_ids[0]) but found one of them on device: cuda:2 I have 4 GPUs (0,1,2,3) and I want to run one Jupyter notebook on GPU 2 and another one on GPU 0. Thus, after executing,
```
export CUDA_VISIBLE_DEVICES=0,1,2,3
```
for the GPU 2 notebook I do,
```
device = torch.device( f'cuda:{2}' if torch.cuda.is_available() else 'cpu')
device, torch.cuda.device_count(), torch.cuda.is_available(), torch.cuda.current_device(), torch.cuda.get_device_properties(1)
```
and after creating a new model or loading one,
```
model = nn.DataParallel( model, device_ids = [ 0, 1, 2, 3])
model = model.to( device)
```
Then, when I start training the model, I get,
```
RuntimeError                              Traceback (most recent call last)
<ipython-input-18-849ffcb53e16> in <module>
 46             with torch.set_grad_enabled( phase == 'train'):
 47                 # [N, Nclass, H, W]
 ---> 48                 prediction = model(X)
 49                 # print( prediction.shape, y.shape)
 50                 loss_matrix = criterion( prediction, y)

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
491             result = self._slow_forward(*input, **kwargs)
492         else:
--> 493             result = self.forward(*input, **kwargs)
494         for hook in self._forward_hooks.values():
495             hook_result = hook(self, input, result)

~/.local/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
144                 raise RuntimeError("module must have its parameters and buffers "
145                                    "on device {} (device_ids[0]) but found one of "
--> 146                                    "them on device: {}".format(self.src_device_obj, t.device))
147
148         inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)

RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:2
```

Response:
DataParallel requires every input tensor be provided on the first device in its device_ids list. It basically uses that device as a staging area before scattering to the other GPUs and it's the device where final outputs are gathered before returning from forward. If you want device 2 to be the primary device then you just need to put it at the front of the list as follows
```
model = nn.DataParallel(model, device_ids = [2, 0, 1, 3])
model.to(f'cuda:{model.device_ids[0]}')
```
After which all tensors provided to model should be on the first device as well.
```
x = ... # input tensor
x = x.to(f'cuda:{model.device_ids[0]}')
y = model(x)
```


Question:
PyTorch [1 if x > 0.5 else 0 for x in outputs ] with tensors I have a list outputs from a sigmoid function as a tensor in PyTorch E.g
```
output (type) = torch.Size([4]) tensor([0.4481, 0.4014, 0.5820, 0.2877], device='cuda:0',
```
As I'm doing binary classification I want to turn all values bellow 0.5 to 0 and above 0.5 to 1. Traditionally with a NumPy array you can use list iterators:
```
output_prediction = [1 if x > 0.5 else 0 for x in outputs ]
```
This would work, however I have to later convert output_prediction back to a tensor to use
```
torch.sum(ouput_prediction == labels.data)
```
Where labels.data is a binary tensor of labels. Is there a way to use list iterators with tensors?

Response:
```py
prob = torch.tensor([0.3,0.4,0.6,0.7])

out = (prob>0.5).float()
# tensor([0.,0.,1.,1.])
```
Explanation: In pytorch, you can directly use prob>0.5 to get a torch.bool type tensor. Then you can convert to float type via .float().


Question:
Creating a torch tensor from a generator I attempt to construct a tensor from a generator as follows:
```

```python
torch.tensor(i**2 for i in range(10))
#Output
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#RuntimeError: Could not infer dtype of generator
#```
#Currently I just do:
#```
```

```python
torch.tensor([i**2 for i in range(10)])
#Output
#tensor([ 0,  1,  4,  9, 16, 25, 36, 49, 64, 81])
#```
#Is there a way to avoid needing this intermediate list?
```

Response:
As @blue-phoenox already points out, it is preferred to use the built-in PyTorch functions to create the tensor directly. But if you have to deal with generator, it can be advisable to use numpy as a intermediate stage. Since PyTorch avoid to copy the numpy array, it should be quite performat (compared to the simple list comprehension)
```

```python
import torch
```

```python
import numpy as np
```

```python
torch.from_numpy(np.fromiter((i**2 for i in range(10)), int))
#Output
#tensor([ 0,  1,  4,  9, 16, 25, 36, 49, 64, 81])
#```
```


Question:
Can I slice tensors with logical indexing or lists of indices? I'm trying to slice a PyTorch tensor using a logical index on the columns. I want the columns that correspond to a 1 value in the index vector. Both slicing and logical indexing are possible, but are they possible together? If so, how? My attempt keeps throwing the unhelpful error TypeError: indexing a tensor with an object of type ByteTensor. The only supported types are integers, slices, numpy scalars and torch.LongTensor or torch.ByteTensor as the only argument. MCVE Desired Output
```
import torch

C = torch.LongTensor([[1, 3], [4, 6]])
# 1 3
# 4 6
```
Logical indexing on the columns only:
```
A_log = torch.ByteTensor([1, 0, 1]) # the logical index
B = torch.LongTensor([[1, 2, 3], [4, 5, 6]])
C = B[:, A_log] # Throws error
```
If the vectors are the same size, logical indexing works:
```
B_truncated = torch.LongTensor([1, 2, 3])
C = B_truncated[A_log]
```
And I can get the desired result by repeating the logical index so that it has the same size as the tensor I am indexing, but then I also have to reshape the output.
```
C = B[A_log.repeat(2, 1)] # [torch.LongTensor of size 4]
C = C.resize_(2, 2)
```
I also tried using a list of indices:
```
A_idx = torch.LongTensor([0, 2]) # the index vector
C = B[:, A_idx] # Throws error
```
If I want contiguous ranges of indices, slicing works:
```
C = B[:, 1:2]
```

Response:
I think this is implemented as the index_select function, you can try
```
import torch

A_idx = torch.LongTensor([0, 2]) # the index vector
B = torch.LongTensor([[1, 2, 3], [4, 5, 6]])
C = B.index_select(1, A_idx)
# 1 3
# 4 6
```


Question:
Understanding input shape to PyTorch conv1D? This seems to be one of the common questions on here (1, 2, 3), but I am still struggling to define the right shape for input to PyTorch conv1D. I have text sequences of length 512 (number of tokens per sequence) with each token being represented by a vector of length 768 (embedding). The batch size I am using is 6. So my input tensor to conv1D is of shape [6, 512, 768].
```
input = torch.randn(6, 512, 768)
```
Now, I want to convolve over the length of my sequence (512) with a kernel size of 2 using the conv1D layer from PyTorch. Understanding 1: I assumed that "in_channels" are the embedding dimension of the conv1D layer. If so, then a conv1D layer will be defined in this way where
```
in_channels = embedding dimension (768)
out_channels = 100 (arbitrary number)
kernel = 2

convolution_layer = nn.conv1D(768, 100, 2)
feature_map = convolution_layer(input)
```
But with this assumption, I get the following error:
```
RuntimeError: Given groups=1, weight of size 100 768 2, expected input `[4, 512, 768]` to have 768 channels, but got 512 channels instead
```
Understanding 2: Then I assumed that "in_channels" is the sequence length of the input sequence. If so, then a conv1D layer will be defined in this way where
```
in_channels = sequence length (512)
out_channels = 100 (arbitrary number)
kernel = 2

convolution_layer = nn.conv1D(512, 100, 2)
feature_map = convolution_layer(input)
```
This works fine and I get an output feature map of dimension [batch_size, 100, 767]. However, I am confused. Shouldn't the convolutional layer convolve over the sequence length of 512 and output a feature map of dimension [batch_size, 100, 511]? I will be really grateful for your help.

Response:
In pytorch your input shape of [6, 512, 768] should actually be [6, 768, 512] where the feature length is represented by the channel dimension and sequence length is the length dimension. Then you can define your conv1d with in/out channels of 768 and 100 respectively to get an output of [6, 100, 511]. Given an input of shape [6, 512, 768] you can convert it to the correct shape with Tensor.transpose.
```
input = input.transpose(1, 2).contiguous()
```
The .contiguous() ensures the memory of the tensor is stored contiguously which helps avoid potential issues during processing.


Question:
How to convert torch tensor to pandas dataframe? I'd like to convert a torch tensor to pandas dataframe but by using pd.DataFrame I'm getting a dataframe filled with tensors instead of numeric values.
```py
import torch
import pandas as  pd
x = torch.rand(4,4)
px = pd.DataFrame(x)
```
Here's what I get when clicking on px in the variable explorer:
```
0   1   2   3
tensor(0.3880)  tensor(0.4598)  tensor(0.4239)  tensor(0.7376)
tensor(0.4174)  tensor(0.9581)  tensor(0.0987)  tensor(0.6359)
tensor(0.6199)  tensor(0.8235)  tensor(0.9947)  tensor(0.9679)
tensor(0.7164)  tensor(0.9270)  tensor(0.7853)  tensor(0.6921)
```

Response:
I found one possible way by converting torch first to numpy:
```
import torch
import pandas as  pd

x = torch.rand(4,4)
px = pd.DataFrame(x.numpy())
```


Question: what the difference between att_mask and key_padding_mask in MultiHeadAttnetion What the difference between att_mask and key_padding_mask in MultiHeadAttnetion of pytorch: key_padding_mask – if provided, specified padding elements in the key will be ignored by the attention. When given a binary mask and a value is True, the corresponding value on the attention layer will be ignored. When given a byte mask and a value is non-zero, the corresponding value on the attention layer will be ignored attn_mask – 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all the batches while a 3D mask allows to specify a different mask for the entries of each batch. Thanks in advance.

Response: The key_padding_mask is used to mask out positions that are padding, i.e., after the end of the input sequence. This is always specific to the input batch and depends on how long are the sequence in the batch compared to the longest one. It is a 2D tensor of shape batch size × input length. On the other hand, attn_mask says what key-value pairs are valid. In a Transformer decoder, a triangle mask is used to simulate the inference time and prevent the attending to the "future" positions. This is what att_mask is usually used for. If it is a 2D tensor, the shape is input length × input length. You can also have a mask that is specific to every item in a batch. In that case, you can use a 3D tensor of shape (batch size × num heads) × input length × input length. (So, in theory, you can simulate key_padding_mask with a 3D att_mask.)


Question:
Torch sum a tensor along an axis How do I sum over the columns of a tensor?
```
torch.Size([10, 100])    --->    torch.Size([10])
```

Response:
The simplest and best solution is to use torch.sum(). To sum all elements of a tensor:
```
torch.sum(x) # gives back a scalar
```
To sum over all rows (i.e. for each column):
```
torch.sum(x, dim=0) # size = [ncol]
```
To sum over all columns (i.e. for each row):
```
torch.sum(x, dim=1) # size = [nrow]
```
It should be noted that the dimension summed over is eliminated from the resulting tensor.


Question:
Pytorch RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor' I am using pytorch for training models. But I got an runtime error when it was computing the cross-entropy loss.
```
Traceback (most recent call last):
  File "deparser.py", line 402, in <module>
    d.train()
  File "deparser.py", line 331, in train
    total, correct, avgloss = self.train_util()
  File "deparser.py", line 362, in train_util
    loss = self.step(X_train, Y_train, correct, total)
  File "deparser.py", line 214, in step
    loss = nn.CrossEntropyLoss()(out.long(), y)
  File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/modules/loss.py", line 862, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/functional.py", line 1550, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/functional.py", line 975, in log_softmax
    return input.log_softmax(dim)
RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'
```
I think this is because the .cuda() function or conversion between torch.Float and torch.Long. But I have tried many ways to change the variable by .cpu()/.cuda() and .long()/.float(), but it still not work. This error message can't be found when searching it on google. Can anyone helps me? Thanks!!! This is the code cause error:
```
def step(self, x, y, correct, total):
    self.optimizer.zero_grad()
    out = self.forward(*x)
    loss = nn.CrossEntropyLoss()(out.long(), y)
    loss.backward()
    self.optimizer.step()
    _, predicted = torch.max(out.data, 1)
    total += y.size(0)
    correct += int((predicted == y).sum().data)
    return loss.data
```
And this function step() is called by:
```
def train_util(self):
    total = 0
    correct = 0
    avgloss = 0
    for i in range(self.step_num_per_epoch):
        X_train, Y_train = self.trainloader()
        self.optimizer.zero_grad()
        if torch.cuda.is_available():
            self.cuda()
            for i in range(len(X_train)):
                X_train[i] = Variable(torch.from_numpy(X_train[i]))
                X_train[i].requires_grad = False
                X_train[i] = X_train[i].cuda()
            Y_train = torch.from_numpy(Y_train)
            Y_train.requires_grad = False
            Y_train = Y_train.cuda()
        loss = self.step(X_train, Y_train, correct, total)
        avgloss+=float(loss)*Y_train.size(0)
        self.optimizer.step()
        if i%100==99:
            print('STEP %d, Loss: %.4f, Acc: %.4f'%(i+1,loss,correct/total))

    return total, correct, avgloss/self.data_len
```
The input data X_train, Y_train = self.trainloader() are numpy arrays at begining. This is a data sample:
```

```python
X_train, Y_train = d.trainloader()
```

```python
X_train[0].dtype
#Output
#dtype('int64')
```

```python
X_train[1].dtype
#Output
#dtype('int64')
```

```python
X_train[2].dtype
#Output
#dtype('int64')
```

```python
Y_train.dtype
#Output
#dtype('float32')
```

```python
X_train[0]
#Output
#array([[   0,    6,    0, ...,    0,    0,    0],
#       [   0, 1944, 8168, ...,    0,    0,    0],
#       [   0,  815,  317, ...,    0,    0,    0],
#       ...,
#       [   0,    0,    0, ...,    0,    0,    0],
#       [   0,   23,    6, ...,    0,    0,    0],
#       [   0,    0,  297, ...,    0,    0,    0]])
```

```python
X_train[1]
#Output
#array([ 6,  7,  8, 21,  2, 34,  3,  4, 19, 14, 15,  2, 13,  3, 11, 22,  4,
#   13, 34, 10, 13,  3, 48, 18, 16, 19, 16, 17, 48,  3,  3, 13])
```

```python
X_train[2]
#Output
#array([ 4,  5,  8, 36,  2, 33,  5,  3, 17, 16, 11,  0,  9,  3, 10, 20,  1,
#   14, 33, 25, 19,  1, 46, 17, 14, 24, 15, 15, 51,  2,  1, 14])
```

```python
Y_train
#Output
#array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
#        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
#       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
#        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
#       ...,
#       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
#        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]],
#      dtype=float32)
#```
#Try all possible combinations: case 1: loss = nn.CrossEntropyLoss()(out, y) I get: RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target' case 2: loss = nn.CrossEntropyLoss()(out.long(), y) as description above case 3: loss = nn.CrossEntropyLoss()(out.float(), y) I get: RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target' case 4: loss = nn.CrossEntropyLoss()(out, y.long()) I get: RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15 case 5: loss = nn.CrossEntropyLoss()(out.long(), y.long()) I get: RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor' case 6: loss = nn.CrossEntropyLoss()(out.float(), y.long()) I get: RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15 case 7: loss = nn.CrossEntropyLoss()(out, y.float()) I get: RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target' case 8: loss = nn.CrossEntropyLoss()(out.long(), y.float()) I get: RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor' case 9: loss = nn.CrossEntropyLoss()(out.float(), y.float()) I get: RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target'
```

Response: I know where the problem is. y should be in torch.int64 dtype without one-hot encoding. And CrossEntropyLoss() will auto encoding it with one-hot (while out is the probability distribution of prediction like one-hot format). It can run now!


Question: Pytorch tensor to change dimension I have a RGB image tensor as (3,H,W), but the plt.imshow() can not show RGB image with this shape. I want to change the tensor to (H,W,3). How can I do that, is pytorch function .view() can do that?

Response:
Find the method. use pytorch permute() method, see details: https://www.geeksforgeeks.org/python-pytorch-permute-method/ code:
```
image.permute(1, 2, 0)
```


Question:
What does "RuntimeError: CUDA error: device-side assert triggered" in PyTorch mean? I have seen a lot of specific posts to particular case-specific problems, but no fundamental motivating explanation. What does this error:
```
RuntimeError: CUDA error: device-side assert triggered
```
mean? Specifically, what is the assert that is being triggered, why is the assert there, and how do we work backwards to debug the problem? As-is, this error message is near useless in diagnosing any problem because of the generality that it seems to say "some code somewhere that touches the GPU" has a problem. The documentation of Cuda also does not seem helpful in this regard, though I could be wrong. https://docs.nvidia.com/cuda/cuda-gdb/index.html

Response: When a device-side error is detected while CUDA device code is running, that error is reported via the usual CUDA runtime API error reporting mechanism. The usual detected error in device code would be something like an illegal address (e.g. attempt to dereference an invalid pointer) but another type is a device-side assert. This type of error is generated whenever a C/C++ assert() occurs in device code, and the assert condition is false. Such an error occurs as a result of a specific kernel. Runtime error checking in CUDA is necessarily asynchronous, but there are probably at least 3 possible methods to start to debug this. Modify the source code to effectively convert asynchronous kernel launches to synchronous kernel launches, and do rigorous error-checking after each kernel launch. This will identify the specific kernel that has caused the error. At that point it may be sufficient simply to look at the various asserts in that kernel code, but you could also use step 2 or 3 below. Run your code with cuda-memcheck. This is a tool something like "valgrind for device code". When you run your code with cuda-memcheck, it will tend to run much more slowly, but the runtime error reporting will be enhanced. It is also usually preferable to compile your code with -lineinfo. In that scenario, when a device-side assert is triggered, cuda-memcheck will report the source code line number where the assert is, and also the assert itself and the condition that was false. You can see here for a walkthrough of using it (albeit with an illegal address error instead of assert(), but the process with assert() will be similar. It should also be possible to use a debugger. If you use a debugger such as cuda-gdb (e.g. on linux) then the debugger will have back-trace reports that will indicate which line the assert was, when it was hit. Both cuda-memcheck and the debugger can be used if the CUDA code is launched from a python script. At this point you have discovered what the assert is and where in the source code it is. Why it is there cannot be answered generically. This will depend on the developers intention, and if it is not commented or otherwise obvious, you will need some method to intuit that somehow. The question of "how to work backwards" is also a general debugging question, not specific to CUDA. You can use printf in CUDA kernel code, and also a debugger like cuda-gdb to assist with this (for example, set a breakpoint prior to the assert, and inspect machine state - e.g. variables - when the assert is about to be hit). With newer GPUs, instead of cuda-memcheck you will probably want to use compute-sanitizer. It works in a similar fashion.


Question:
Is this the way to create a PyTorch scalar? I'm new to PyTorch, and just wanted to kindly confirm if the following creates scalars with values, 1, 2, and 3, respectively?
```
import torch

a = torch.tensor(1)
b = torch.tensor(2)
c = torch.tensor(3)
```
Thanks.

Response:
From the example in the documentation for torch.tensor:
```

```python
torch.tensor(3.14159)  # Create a scalar (zero-dimensional tensor)
#Output
#tensor(3.1416)
#```
#Therefore, it appears they endorse that passing a number creates the corresponding scalar.
```


Question:
How to fix 'Input and hidden tensors are not at the same device' in pytorch When I want to put the model on the GPU, I get the following error: "RuntimeError: Input and hidden tensors are not at the same device, found input tensor at cuda:0 and hidden tensor at cpu" However, all of the above had been put on the GPU:
```
for m in model.parameters():
    print(m.device) #return cuda:0
```
```py
if torch.cuda.is_available():
    model = model.cuda()
    test = test.cuda() # test is the Input
```
Windows 10 server Pytorch 1.2.0 + cuda 9.2 cuda 9.2 cudnn 7.6.3 for cuda 9.2

Response:
You need to move the model, the inputs, and the targets to Cuda:
```
if torch.cuda.is_available():
   model.cuda()
   inputs = inputs.cuda()
   target = target.cuda()
```


Question: Difference between Parameter vs. Tensor in PyTorch I would like to know the difference between PyTorch Parameter and Tensor? The existing answer is for the old PyTorch where variables are being used?

Response:
This is the whole idea of the Parameter class (attached) in a single image. Since it is sub-classed from Tensor it is a Tensor. But there is a trick. Parameters that are inside of a module are added to the list of Module parameters. If m is your module m.parameters() will hold your parameter. Here is the example:
```
class M(nn.Module):
    def __init__(self):
        super().__init__()
        self.weights = nn.Parameter(torch.randn(2, 2))
        self.bias = nn.Parameter(torch.zeros(2))

    def forward(self, x):
        return x @ self.weights + self.bias

m=M()
m.parameters()
list(m.parameters())

---

[Parameter containing:
 tensor([[ 0.5527,  0.7096],
         [-0.2345, -1.2346]], requires_grad=True), Parameter containing:
 tensor([0., 0.], requires_grad=True)]
```
You see how the parameters will show what we defined. And if we just add a tensor inside a class, like self.t = Tensor, it will not show in the parameters list. That is literally it. Nothing fancy.


Question:
Pytorch softmax: What dimension to use? The function torch.nn.functional.softmax takes two parameters: input and dim. According to its documentation, the softmax operation is applied to all slices of input along the specified dim, and will rescale them so that the elements lie in the range (0, 1) and sum to 1. Let input be:
```
input = torch.randn((3, 4, 5, 6))
```
Suppose I want the following, so that every entry in that array is 1:
```
sum = torch.sum(input, dim = 3) # sum's size is (3, 4, 5, 1)
```
How should I apply softmax?
```
softmax(input, dim = 0) # Way Number 0
softmax(input, dim = 1) # Way Number 1
softmax(input, dim = 2) # Way Number 2
softmax(input, dim = 3) # Way Number 3
```
My intuition tells me that is the last one, but I am not sure. English is not my first language and the use of the word along seemed confusing to me because of that. I am not very clear on what "along" means, so I will use an example that could clarify things. Suppose we have a tensor of size (s1, s2, s3, s4), and I want this to happen

Response:
Steven's answer is not correct. See the snapshot below. It is actually the reverse way. Image transcribed as code:
```

```python
x = torch.tensor([[1,2],[3,4]],dtype=torch.float)
```

```python
F.softmax(x,dim=0)
#Output
#tensor([[0.1192, 0.1192],
#        [0.8808, 0.8808]])
```

```python
F.softmax(x,dim=1)
#Output
#tensor([[0.2689, 0.7311],
#        [0.2689, 0.7311]])
#```
```


Question:
What does the .numpy() function do? I tried searching for the documentation online but I can't find anything that gives me an answer. What does .numpy() function do? The example code given is:
```
y_true = []
for X_batch, y_batch in mnist_test:
    y_true.append(y_batch.numpy()[0].tolist())
```

Response: Both in Pytorch and Tensorflow, the .numpy() method is pretty much straightforward. It converts a tensor object into an numpy.ndarray object. This implicitly means that the converted tensor will be now processed on the CPU.


Question: How to ignore and initialize Missing key(s) in state_dict My saved state_dict does not contain all the layers that are in my model. How can I ignore the Missing key(s) in state_dict error and initialize the remaining weights?

Response:
This can be achieved by passing strict=False to load_state_dict.
```
load_state_dict(state_dict, strict=False)
```
Documentation


Question: Why do we need to explicitly call zero_grad()? [duplicate] This question already has answers here: Why do we need to call zero_grad() in PyTorch? (9 answers) Closed 5 years ago. Why do we need to explicitly zero the gradients in PyTorch? Why can't gradients be zeroed when loss.backward() is called? What scenario is served by keeping the gradients on the graph and asking the user to explicitly zero the gradients?

Response: We explicitly need to call zero_grad() because, after loss.backward() (when gradients are computed), we need to use optimizer.step() to proceed gradient descent. More specifically, the gradients are not automatically zeroed because these two operations, loss.backward() and optimizer.step(), are separated, and optimizer.step() requires the just computed gradients. In addition, sometimes, we need to accumulate gradient among some batches; to do that, we can simply call backward multiple times and optimize once.


Question: What does `gather()` do in PyTorch in layman terms? What does torch.gather() do? This answer is hard to understand.

Response: torch.gather creates a new tensor from the input tensor by taking the values from each row along the input dimension dim. The values in torch.LongTensor, passed as index, specify which value to take from each 'row'. The dimension of the output tensor is same as the dimension of index tensor. Following illustration from the official docs explains it more clearly: (Note: In the illustration, indexing starts from 1 and not 0). In first example, the dimension given is along rows (top to bottom), so for (1,1) position of result, it takes row value from the index for the src that is 1. At (1,1) in source value is 1 so, outputs 1 at (1,1) in result. Similarly for (2,2) the row value from the index for src is 3. At (3,2) the value in src is 8 and hence outputs 8 and so on. Similarly for second example, indexing is along columns, and hence at (2,2) position of the result, the column value from the index for src is 3, so at (2,3) from src ,6 is taken and outputs to result at (2,2)


Question:
Why torch.sum() before doing .backward()? I can see what this code below from this video is trying to do. But the sum from y=torch.sum(x**2) confuses me. With sum operation, y becomes a tensor with one single value. As I understand .backward() as calculating derivatives, why would we want to use sum and reduce y to one value?
```
import pytorch
import matplotlib.pyplot as plt
x = torch.linspace(-10.0,10.0,10, requires_grad=True)
Y = x**2
y = torch.sum(x**2)
y.backward()

plt.plot(x.detach().numpy(), Y.detach().numpy(), label="Y")
plt.plot(x.detach().numpy(), x.grad.detach().numpy(), label="derivatives")
plt.legend()
```

Response: You can only compute partial derivatives for a scalar function. What backwards() gives you is d loss/d parameter and you expect a single gradient value per parameter/variable. Had your loss function been a vector function, i.e., mapping from multiple inputs to multiple outputs, you would have ended up with multiple gradients per parameter/variable. Please see this answer for more information.


Question: Fixing the seed for torch random_split() Is it possible to fix the seed for torch.utils.data.random_split() when splitting a dataset so that it is possible to reproduce the test results?

Response:
EDIT: This answer, while still factually correct, is now obsolete. See comment below and an answer by Matteo Pennisi You can use torch.manual_seed function to seed the script globally:
```
import torch
torch.manual_seed(0)
```
See reproducibility documentation for more information. If you want to specifically seed torch.utils.data.random_split you could "reset" the seed to it's initial value afterwards. Simply use torch.initial_seed() like this:
```
torch.manual_seed(torch.initial_seed())
```
AFAIK pytorch does not provide arguments like seed or random_state (which could be seen in sklearn for example).


Question:
What does `-1` of `view()` mean in PyTorch? As the question says, what does -1 of view() do in PyTorch?
```py

```python
a = torch.arange(1, 17)
```

```python
a
#Output
#tensor([  1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,
#         11.,  12.,  13.,  14.,  15.,  16.])
```

```python
a.view(1,-1)
#Output
#tensor([[  1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,
#          11.,  12.,  13.,  14.,  15.,  16.]])
```

```python
a.view(-1,1)
#Output
#tensor([[  1.],
#        [  2.],
#        [  3.],
#        [  4.],
#        [  5.],
#        [  6.],
#        [  7.],
#        [  8.],
#        [  9.],
#        [ 10.],
#        [ 11.],
#        [ 12.],
#        [ 13.],
#        [ 14.],
#        [ 15.],
#        [ 16.]])
#```
#Does -1 of view() in PyTorch generate additional dimension? Does -1 of view() in PyTorch behave the same as -1 of reshape() in NumPy?
```

Response:
Yes, it does behave like -1 in numpy.reshape(), i.e. the actual value for this dimension will be inferred so that the number of elements in the view matches the original number of elements. For instance:
```py
import torch

x = torch.arange(6)

print(x.view(3, -1))      # inferred size will be 2 as 6 / 3 = 2
# tensor([[ 0.,  1.],
#         [ 2.,  3.],
#         [ 4.,  5.]])

print(x.view(-1, 6))      # inferred size will be 1 as 6 / 6 = 1
# tensor([[ 0.,  1.,  2.,  3.,  4.,  5.]])

print(x.view(1, -1, 2))   # inferred size will be 3 as 6 / (1 * 2) = 3
# tensor([[[ 0.,  1.],
#          [ 2.,  3.],
#          [ 4.,  5.]]])

# print(x.view(-1, 5))    # throw error as there's no int N so that 5 * N = 6
# RuntimeError: invalid argument 2: size '[-1 x 5]' is invalid for input with 6 elements

print(x.view(-1, -1, 3))  # throw error as only one dimension can be inferred
# RuntimeError: invalid argument 1: only one dimension can be inferred
```


Question: When should I use nn.ModuleList and when should I use nn.Sequential? I am new to Pytorch and one thing that I don't quite understand is the usage of nn.ModuleList and nn.Sequential. Can I know when I should use one over the other? Thanks.

Response: nn.ModuleList does not have a forward method, but nn.Sequential does have one. So you can wrap several modules in nn.Sequential and run it on the input. nn.ModuleList is just a Python list (though it's useful since the parameters can be discovered and trained via an optimizer). While nn.Sequential is a module that sequentially runs the component on the input.


Question:
Pytorch: How can I find indices of first nonzero element in each row of a 2D tensor? I have a 2D tensor with some nonzero element in each row like this:
```
import torch
tmp = torch.tensor([[0, 0, 1, 0, 1, 0, 0],
                    [0, 0, 0, 1, 1, 0, 0]], dtype=torch.float)
```
I want a tensor containing the index of first nonzero element in each row:
```
indices = tensor([2],
                 [3])
```
How can I calculate it in Pytorch?

Response:
I have simplified Iman's approach to do the following:
```
idx = torch.arange(tmp.shape[1], 0, -1)
tmp2= tmp * idx
indices = torch.argmax(tmp2, 1, keepdim=True)
```


Question:
PyTorch - How to get learning rate during training? While training, I'd like to know the value of learning_rate. What should I do? It's my code, like this:
```
my_optimizer = torch.optim.SGD(my_model.parameters(),
                               lr=0.001,
                               momentum=0.99,
                               weight_decay=2e-3)
```
Thank you.

Response:
For only one parameter group like in the example you've given, you can use this function and call it during training to get the current learning rate:
```
def get_lr(optimizer):
    for param_group in optimizer.param_groups:
        return param_group['lr']
```


Question:
Is GradScaler necessary with Mixed precision training with pytorch? So going the AMP: Automatic Mixed Precision Training tutorial for Normal networks, I found out that there are two versions, Automatic and GradScaler. I just want to know if it's advisable / necessary to use the GradScaler with the training becayse it is written in the document that: Gradient scaling helps prevent gradients with small magnitudes from flushing to zero (“underflowing”) when training with mixed precision.
```
scaler = torch.cuda.amp.GradScaler()
for epoch in range(1):
    for input, target in zip(data, targets):
        with torch.cuda.amp.autocast():
            output = net(input)
            loss = loss_fn(output, target)

        scaler.scale(loss).backward()
        scaler.step(opt)
        scaler.update()
        opt.zero_grad()
```
Also, looking at NVIDIA Apex Documentation for PyTorch, they have used it as,
```
from apex import amp

model, optimizer = amp.initialize(model, optimizer)

loss = criterion(…)
with amp.scale_loss(loss, optimizer) as scaled_loss:
    scaled_loss.backward()
optimizer.step()
```
I think this is what GradScaler does too so I think it is a must. Can someone help me with the query here.

Response: Short answer: yes, your model may fail to converge without GradScaler(). There are three basic problems with using FP16: Weight updates: with half precision, 1 + 0.0001 rounds to 1. autocast() takes care of this one. Vanishing gradients: with half precision, anything less than (roughly) 2e-14 rounds to 0, as opposed to single precision 2e-126. GradScaler() takes care of this one. Explosive loss: similar to the above, overflow is also much more likely with half precision. This is also managed by autocast() context.


Question: Check the total number of parameters in a PyTorch model How do I count the total number of parameters in a PyTorch model? Something similar to model.count_params() in Keras.

Response:
PyTorch doesn't have a function to calculate the total number of parameters as Keras does, but it's possible to sum the number of elements for every parameter group:
```
pytorch_total_params = sum(p.numel() for p in model.parameters())
```
If you want to calculate only the trainable parameters:
```
pytorch_total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
```
Answer inspired by this answer on PyTorch Forums.


Question:
How to count the amount of layers in a CNN? The Pytorch implementation of ResNet-18. has the following structure, which appears to be 54 layers, not 18. So why is it called "18"? How many layers does it actually have?
```py
ResNet (
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
  (relu): ReLU (inplace)
  (maxpool): MaxPool2d (size=(3, 3), stride=(2, 2), padding=(1, 1), dilation=(1, 1))
  (layer1): Sequential (
    (0): BasicBlock (
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
    )
    (1): BasicBlock (
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
    )
  )
  (layer2): Sequential (
    (0): BasicBlock (
      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
      (downsample): Sequential (
        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
      )
    )
    (1): BasicBlock (
      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
    )
  )
  (layer3): Sequential (
    (0): BasicBlock (
      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
      (downsample): Sequential (
        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
      )
    )
    (1): BasicBlock (
      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
    )
  )
  (layer4): Sequential (
    (0): BasicBlock (
      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
      (downsample): Sequential (
        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
      )
    )
    (1): BasicBlock (
      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
    )
  )
  (avgpool): AvgPool2d (
  )
  (fc): Linear (512 -> 1000)
)
```

Response: From your output, we can know that there are 20 convolution layers (one 7x7 conv, 16 3x3 conv, and plus 3 1x1 conv for downsample). Basically, if you ignore the 1x1 conv, and counting the FC (linear) layer, the number of layers are 18. And I've also made an example on how to visualize your architecture in pytorch via graphviz, hope it will help you understand your architecture.