Question:
How to flatten a tensor in PyTorch? Given a tensor of multiple dimensions, how do I flatten it so that it has a single dimension?
```
torch.Size([2, 3, 5])    ⟶ flatten ⟶    torch.Size([30])
```

Response:
TL;DR: torch.flatten() Use torch.flatten() which was introduced in v0.4.1 and documented in v1.0rc1:
```

```python
t = torch.tensor([[[1, 2],
#Output
#                       [3, 4]],
#                      [[5, 6],
#                       [7, 8]]])
```

```python
torch.flatten(t)
#Output
#tensor([1, 2, 3, 4, 5, 6, 7, 8])
```

```python
torch.flatten(t, start_dim=1)
#Output
#tensor([[1, 2, 3, 4],
#        [5, 6, 7, 8]])
#```
#For v0.4.1 and earlier, use t.reshape(-1). With t.reshape(-1): If the requested view is contiguous in memory this will equivalent to t.view(-1) and memory will not be copied. Otherwise it will be equivalent to t.contiguous().view(-1). Other non-options: t.view(-1) won't copy memory, but may not work depending on original size and stride t.resize(-1) gives RuntimeError (see below) t.resize(t.numel()) warning about being a low-level method (see discussion below) (Note: pytorch's reshape() may change data but numpy's reshape() won't.) t.resize(t.numel()) needs some discussion. The torch.Tensor.resize_ documentation says: The storage is reinterpreted as C-contiguous, ignoring the current strides (unless the target size equals the current size, in which case the tensor is left unchanged) Given the current strides will be ignored with the new (1, numel()) size, the order of the elements may apppear in a different order than with reshape(-1). However, "size" may mean the memory size, rather than the tensor's size. It would be nice if t.resize(-1) worked for both convenience and efficiency, but with torch 1.0.1.post2, t = torch.rand([2, 3, 5]); t.resize(-1) gives:
#```
#RuntimeError: requested resize to -1 (-1 elements in total), but the given
#tensor has a size of 2x2 (4 elements). autograd's resize can only change the
#shape of a given tensor, while preserving the number of elements.
#```
#I raised a feature request for this here, but the consensus was that resize() was a low level method, and reshape() should be used in preference.
```


Question:
Random Choice with Pytorch? I have a tensor of pictures, and would like to randomly select from it. I'm looking for the equivalent of np.random.choice().
```
import torch

pictures = torch.randint(0, 256, (1000, 28, 28, 3))
```
Let's say I want 10 of these pictures.

Response:
torch has no equivalent implementation of np.random.choice(), see the discussion here. The alternative is indexing with a shuffled index or random integers. To do it with replacement: Generate n random indices Index your original tensor with these indices
```
pictures[torch.randint(len(pictures), (10,))]
```
To do it without replacement: Shuffle the index Take the n first elements
```
indices = torch.randperm(len(pictures))[:10]

pictures[indices]
```
Read more about torch.randint and torch.randperm. Second code snippet is inspired by this post in PyTorch Forums.


Question: Get single random example from PyTorch DataLoader How do I get a single random example from a PyTorch DataLoader? If my DataLoader gives minbatches of multiple images and labels, how do I get a single random image and label? Note that I don't want a single image and label per minibatch, I want a total of one example.

Response:
If your DataLoader is something like this:
```py
test_loader = DataLoader(image_datasets['val'], batch_size=batch_size, shuffle=True)
```
it is giving you a batch of size batch_size, and you can pick out a single random example by directly indexing the batch:
```
for test_images, test_labels in test_loader:
    sample_image = test_images[0]    # Reshape them according to your needs.
    sample_label = test_labels[0]
```
Alternative solutions You can use RandomSampler to obtain random samples. Use a batch_size of 1 in your DataLoader. Directly take samples from your DataSet like so:
```
mnist_test = datasets.MNIST('../MNIST/', train=False, transform=transform)
```
Now use this dataset to take samples:
```
for image, label in mnist_test:
      # do something with image and other attributes
```
(Probably the best) See here:
```
inputs, classes = next(iter(dataloader))
```


Question:
Using torch.nn.DataParallel with a custom CUDA extension To my understanding, the built-in PyTorch operations all automatically handle batches through implicit vectorization, allowing parallelism across multiple GPUs. However, when writing a custom operation in CUDA as per the Documentation, the LLTM example given performs operations that are batch invariant, for example computing the gradient of the Sigmoid function elementwise. However, I have a use case that is not batch element invariant and not vectorizable. Running on a single GPU, I currently (inefficiently) loop over each element in the batch, performing a kernel launch for each, like so (written in the browser, just to demonstrate):
```
std::vector<at::Tensor> op_cuda_forward(at::Tensor input,
                                        at::Tensor elementSpecificParam) {

    auto output = at::zeros(torch::CUDA(/* TYPE */), {/* DIMENSIONS */});

    const size_t blockDim = //
    const size_t gridDim = //
    const size_t = numBatches = //

    for (size_t i = 0; i < numBatches; i++) {
        op_cuda_forward_kernel<T><<<gridDim, blockDim>>>(input[i],
                                                         elementSpecificParam[i],
                                                         output[i]);
    }

    return {output};
}
```
However, I wish to split this operation over multiple GPUs by batch element. How would the allocation of the output Tensor work in a multi-GPU scenario? Of course, one may create intermediate Tensors on each GPU before launching the appropriate kernel, however, the overhead of copying the input data to each GPU and back again would be problematic. Is there a simpler way to launch the kernels without first probing the environment for GPU information (# GPU's etc)? The end goal is to have a CUDA operation that works with torch.nn.DataParallel.

Response:
This is kind of unusual, as commonly "Batch" is exactly defined as all operations of the network being invariant along that dimension. So you could, for example, just introduce another dimension. So you have the "former batch dimension" in which your operation is not invariant. For this keep your current implementation. Then, parallelize over the new dimension of multiple "actual batches" of data. But, to stay closer to the question you asked, I see two options: As you said, inside your implementation figure out which original batch you are operating on (depending on total number of parallel splits, etc). This can become hairy. Consider your parameter as Part of Input! In your outside call, pass the parameter along your input data to the forward of your model. So (Pythonlike-Pseudocode):
```
Network(nn.Module):
  ...
  def forward(x, parameter):
    x=self.pre_modules(x)
    x=self.custom_module(x,parameter)
    return x

parameter=torch.zeros(16,requires_grad=True)
net=nn.DataParallel(model)
net(input,parameter)
```
If your are willing to accept that this will be a leaky abstraction of the network and are mainly interested in getting things to work, I would try out the latter approach first.


Question: How to Display Custom Images in Tensorboard (e.g. Matplotlib Plots)? The Image Dashboard section of the Tensorboard ReadMe says: Since the image dashboard supports arbitrary pngs, you can use this to embed custom visualizations (e.g. matplotlib scatterplots) into TensorBoard. I see how a pyplot image could be written to file, read back in as a tensor, and then used with tf.image_summary() to write it to TensorBoard, but this statement from the readme suggests there is a more direct way. Is there? If so, is there any further documentation and/or examples of how to do this efficiently?

Response:
It is quite easy to do if you have the image in a memory buffer. Below, I show an example, where a pyplot is saved to a buffer and then converted to a TF image representation which is then sent to an image summary.
```
import io
import matplotlib.pyplot as plt
import tensorflow as tf

def gen_plot():
    """Create a pyplot plot and save to buffer."""
    plt.figure()
    plt.plot([1, 2])
    plt.title("test")
    buf = io.BytesIO()
    plt.savefig(buf, format='png')
    buf.seek(0)
    return buf

# Prepare the plot
plot_buf = gen_plot()

# Convert PNG buffer to TF image
image = tf.image.decode_png(plot_buf.getvalue(), channels=4)

# Add the batch dimension
image = tf.expand_dims(image, 0)

# Add image summary
summary_op = tf.summary.image("plot", image)

# Session
with tf.Session() as sess:
    # Run
    summary = sess.run(summary_op)
    # Write summary
    writer = tf.train.SummaryWriter('./logs')
    writer.add_summary(summary)
    writer.close()
```
This gives the following TensorBoard visualization:


Question: How to apply layer-wise learning rate in Pytorch? I know that it is possible to freeze single layers in a network for example to train only the last layers of a pre-trained model. What I’m looking for is a way to apply certain learning rates to different layers. So for example a very low learning rate of 0.000001 for the first layer and then increasing the learning rate gradually for each of the following layers. So that the last layer then ends up with a learning rate of 0.01 or so. Is this possible in pytorch? Any idea how I can archive this?

Response:
Here is the solution:
```
from torch.optim import Adam

model = Net()

optim = Adam(
    [
        {"params": model.fc.parameters(), "lr": 1e-3},
        {"params": model.agroupoflayer.parameters()},
        {"params": model.lastlayer.parameters(), "lr": 4e-2},
    ],
    lr=5e-4,
)
```
Other parameters that are didn't specify in optimizer will not optimize. So you should state all layers or groups(OR the layers you want to optimize). and if you didn't specify the learning rate it will take the global learning rate(5e-4). The trick is when you create the model you should give names to the layers or you can group it.


Question:
How to get an output dimension for each layer of the Neural Network in Pytorch?
```
class Model(nn.Module):
  def __init__(self):
    super(Model, self).__init__()
    self.net = nn.Sequential(
      nn.Conv2d(in_channels = 3, out_channels = 16),
      nn.ReLU(),
      nn.MaxPool2d(2),
      nn.Conv2d(in_channels = 16, out_channels = 16),
      nn.ReLU(),
      Flatten(),
      nn.Linear(4096, 64),
      nn.ReLU(),
      nn.Linear(64, 10))

  def forward(self, x):
    return self.net(x)
```
I have created this model without a firm knowledge in Neural Network and I just fixed parameters until it worked in the training. I am not sure how to get the output dimension for each layer (e.g. output dimension after the first layer). Is there an easy way to do this in Pytorch?

Response:
You can use torchsummary, for instance, for ImageNet dimension(3x224x224):
```
from torchvision import models
from torchsummary import summary

vgg = models.vgg16()
summary(vgg, (3, 224, 224))

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1         [-1, 64, 224, 224]           1,792
              ReLU-2         [-1, 64, 224, 224]               0
            Conv2d-3         [-1, 64, 224, 224]          36,928
              ReLU-4         [-1, 64, 224, 224]               0
         MaxPool2d-5         [-1, 64, 112, 112]               0
            Conv2d-6        [-1, 128, 112, 112]          73,856
              ReLU-7        [-1, 128, 112, 112]               0
            Conv2d-8        [-1, 128, 112, 112]         147,584
              ReLU-9        [-1, 128, 112, 112]               0
        MaxPool2d-10          [-1, 128, 56, 56]               0
           Conv2d-11          [-1, 256, 56, 56]         295,168
             ReLU-12          [-1, 256, 56, 56]               0
           Conv2d-13          [-1, 256, 56, 56]         590,080
             ReLU-14          [-1, 256, 56, 56]               0
           Conv2d-15          [-1, 256, 56, 56]         590,080
             ReLU-16          [-1, 256, 56, 56]               0
        MaxPool2d-17          [-1, 256, 28, 28]               0
           Conv2d-18          [-1, 512, 28, 28]       1,180,160
             ReLU-19          [-1, 512, 28, 28]               0
           Conv2d-20          [-1, 512, 28, 28]       2,359,808
             ReLU-21          [-1, 512, 28, 28]               0
           Conv2d-22          [-1, 512, 28, 28]       2,359,808
             ReLU-23          [-1, 512, 28, 28]               0
        MaxPool2d-24          [-1, 512, 14, 14]               0
           Conv2d-25          [-1, 512, 14, 14]       2,359,808
             ReLU-26          [-1, 512, 14, 14]               0
           Conv2d-27          [-1, 512, 14, 14]       2,359,808
             ReLU-28          [-1, 512, 14, 14]               0
           Conv2d-29          [-1, 512, 14, 14]       2,359,808
             ReLU-30          [-1, 512, 14, 14]               0
        MaxPool2d-31            [-1, 512, 7, 7]               0
           Linear-32                 [-1, 4096]     102,764,544
             ReLU-33                 [-1, 4096]               0
          Dropout-34                 [-1, 4096]               0
           Linear-35                 [-1, 4096]      16,781,312
             ReLU-36                 [-1, 4096]               0
          Dropout-37                 [-1, 4096]               0
           Linear-38                 [-1, 1000]       4,097,000
================================================================
Total params: 138,357,544
Trainable params: 138,357,544
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.57
Forward/backward pass size (MB): 218.59
Params size (MB): 527.79
Estimated Total Size (MB): 746.96
----------------------------------------------------------------
```
Source: model-summary-in-pytorch


Question:
How to use PyTorch multiprocessing? I'm trying to use python's multiprocessing Pool method in pytorch to process a image. Here's the code:
```
from multiprocessing import Process, Pool
from torch.autograd import Variable
import numpy as np
from scipy.ndimage import zoom

def get_pred(args):

  img = args[0]
  scale = args[1]
  scales = args[2]
  img_scale = zoom(img.numpy(),
                     (1., 1., scale, scale),
                     order=1,
                     prefilter=False,
                     mode='nearest')

  # feed input data
  input_img = Variable(torch.from_numpy(img_scale),
                     volatile=True).cuda()
  return input_img

scales = [1,2,3,4,5]
scale_list = []
for scale in scales:
    scale_list.append([img,scale,scales])
multi_pool = Pool(processes=5)
predictions = multi_pool.map(get_pred,scale_list)
multi_pool.close()
multi_pool.join()
```
I'm getting this error:
```
`RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
```
` In this line:
```
predictions = multi_pool.map(get_pred,scale_list)
```
Can anyone tell me what I'm doing wrong ?

Response:
As stated in pytorch documentation the best practice to handle multiprocessing is to use torch.multiprocessing instead of multiprocessing. Be aware that sharing CUDA tensors between processes is supported only in Python 3, either with spawn or forkserver as start method. Without touching your code, a workaround for the error you got is replacing
```
from multiprocessing import Process, Pool
```
with:
```
from torch.multiprocessing import Pool, Process, set_start_method
try:
     set_start_method('spawn')
except RuntimeError:
    pass
```


Question: No N-dimensional tranpose in PyTorch PyTorch's torch.transpose function only transposes 2D inputs. Documentation is here. On the other hand, Tensorflow's tf.transpose function allows you to transpose a tensor of N arbitrary dimensions. Can someone please explain why PyTorch does not/cannot have N-dimension transpose functionality? Is this due to the dynamic nature of the computation graph construction in PyTorch versus Tensorflow's Define-then-Run paradigm?

Response:
It's simply called differently in pytorch. torch.Tensor.permute will allow you to swap dimensions in pytorch like tf.transpose does in TensorFlow. As an example of how you'd convert a 4D image tensor from NHWC to NCHW (not tested, so might contain bugs):
```

```python
img_nhwc = torch.randn(10, 480, 640, 3)
```

```python
img_nhwc.size()
#Output
#torch.Size([10, 480, 640, 3])
```

```python
img_nchw = img_nhwc.permute(0, 3, 1, 2)
```

```python
img_nchw.size()
#Output
#torch.Size([10, 3, 480, 640])
#```
```


Question:
How to construct a network with two inputs in PyTorch Suppose I want to have the general neural network architecture:
```
Input1 --> CNNLayer
                    \
                     ---> FCLayer ---> Output
                    /
Input2 --> FCLayer
```
Input1 is image data, input2 is non-image data. I have implemented this architecture in Tensorflow. All pytorch examples I have found are one input go through each layer. How can I define forward func to process 2 inputs separately then combine them in a middle layer?

Response:
By "combine them" I assume you mean to concatenate the two inputs. Assuming you concat along the second dimension:
```
import torch
from torch import nn

class TwoInputsNet(nn.Module):
  def __init__(self):
    super(TwoInputsNet, self).__init__()
    self.conv = nn.Conv2d( ... )  # set up your layer here
    self.fc1 = nn.Linear( ... )  # set up first FC layer
    self.fc2 = nn.Linear( ... )  # set up the other FC layer

  def forward(self, input1, input2):
    c = self.conv(input1)
    f = self.fc1(input2)
    # now we can reshape `c` and `f` to 2D and concat them
    combined = torch.cat((c.view(c.size(0), -1),
                          f.view(f.size(0), -1)), dim=1)
    out = self.fc2(combined)
    return out
```
Note that when you define the number of inputs to self.fc2 you need to take into account both out_channels of self.conv as well as the output spatial dimensions of c.


Question: PyTorch: What's the difference between state_dict and parameters()? In order to access a model's parameters in pytorch, I saw two methods: using state_dict and using parameters() I wonder what's the difference, or if one is good practice and the other is bad practice. Thanks

Response:
The parameters() only gives the module parameters i.e. weights and biases. Returns an iterator over module parameters. You can check the list of the parameters as follows:
```
for name, param in model.named_parameters():
    if param.requires_grad:
        print(name)
```
On the other hand, state_dict returns a dictionary containing a whole state of the module. Check its source code that contains not just the call to parameters but also buffers, etc. Both parameters and persistent buffers (e.g. running averages) are included. Keys are the corresponding parameter and buffer names. Check all keys that state_dict contains using:
```
model.state_dict().keys()
```
For example, in state_dict, you'll find entries like bn1.running_mean and running_var, which are not present in .parameters(). If you only want to access parameters, you can simply use .parameters(), while for purposes like saving and loading model as in transfer learning, you'll need to save state_dict not just parameters.


Question:
Understanding a simple LSTM pytorch
```
import torch,ipdb
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable

rnn = nn.LSTM(input_size=10, hidden_size=20, num_layers=2)
input = Variable(torch.randn(5, 3, 10))
h0 = Variable(torch.randn(2, 3, 20))
c0 = Variable(torch.randn(2, 3, 20))
output, hn = rnn(input, (h0, c0))
```
This is the LSTM example from the docs. I don't know understand the following things: What is output-size and why is it not specified anywhere? Why does the input have 3 dimensions. What does 5 and 3 represent? What are 2 and 3 in h0 and c0, what do those represent? Edit:
```
import torch,ipdb
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable
import torch.nn.functional as F

num_layers=3
num_hyperparams=4
batch = 1
hidden_size = 20
rnn = nn.LSTM(input_size=num_hyperparams, hidden_size=hidden_size, num_layers=num_layers)

input = Variable(torch.randn(1, batch, num_hyperparams)) # (seq_len, batch, input_size)
h0 = Variable(torch.randn(num_layers, batch, hidden_size)) # (num_layers, batch, hidden_size)
c0 = Variable(torch.randn(num_layers, batch, hidden_size))
output, hn = rnn(input, (h0, c0))
affine1 = nn.Linear(hidden_size, num_hyperparams)

ipdb.set_trace()
print output.size()
print h0.size()
```
*** RuntimeError: matrices expected, got 3D, 2D tensors at

Response:
The output for the LSTM is the output for all the hidden nodes on the final layer. hidden_size - the number of LSTM blocks per layer. input_size - the number of input features per time-step. num_layers - the number of hidden layers. In total there are hidden_size * num_layers LSTM blocks. The input dimensions are (seq_len, batch, input_size). seq_len - the number of time steps in each input stream. batch - the size of each batch of input sequences. The hidden and cell dimensions are: (num_layers, batch, hidden_size) output (seq_len, batch, hidden_size * num_directions): tensor containing the output features (h_t) from the last layer of the RNN, for each t. So there will be hidden_size * num_directions outputs. You didn't initialise the RNN to be bidirectional so num_directions is 1. So output_size = hidden_size. Edit: You can change the number of outputs by using a linear layer:
```
out_rnn, hn = rnn(input, (h0, c0))
lin = nn.Linear(hidden_size, output_size)
v1 = nn.View(seq_len*batch, hidden_size)
v2 = nn.View(seq_len, batch, output_size)
output = v2(lin(v1(out_rnn)))
```
Note: for this answer I assumed that we're only talking about non-bidirectional LSTMs. Source: PyTorch docs.


Question: What is the difference between Tensor.size and Tensor.shape in PyTorch? What is the difference between Tensor.size and Tensor.shape in Pytorch? I want to get the number of elements and the dimensions of Tensor. For example for a tensor with the dimensions of 2 by 3 by 4 I expect 24 for number of elements and (2,3,4) for dimension. Thanks.

Response: .shape is an alias for .size(), and was added to more closely match numpy, see this discussion here.


Question:
RuntimeError: Expected object of type torch.DoubleTensor but found type torch.FloatTensor for argument #2 'weight' My input tensor is torch.DoubleTensor type. But I got the RuntimeError below:
```
RuntimeError: Expected object of type torch.DoubleTensor but found type torch.FloatTensor for argument #2 'weight'
```
I didn't specify the type of the weight explicitly(i.e. I did not init my weight by myself. The weight is created by pytorch). What will influence the type of weight in the forward process? Thanks a lot!!

Response:
The default type for weights and biases are torch.FloatTensor. So, you'll need to cast either your model to torch.DoubleTensor or cast your inputs to torch.FloatTensor. For casting your inputs you can do
```
X = X.float()
```
or cast your complete model to DoubleTensor as
```
model = model.double()
```
You can also set the default type for all tensors using
```
pytorch.set_default_tensor_type('torch.DoubleTensor')
```
It is better to convert your inputs to float rather than converting your model to double, because mathematical computations on double datatype is considerably slower on GPU.


Question: Why is PyTorch called PyTorch? [closed] Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers. This question does not appear to be about programming within the scope defined in the help center. Closed 4 years ago. This post was edited and submitted for review 3 years ago and failed to reopen the post: Original close reason(s) were not resolved Improve this question I have been looking into deep learning frameworks lately and have been wondering about the origin of the name of PyTorch. With Keras, their home page nicely explains the name's origin, and with something like TensorFlow, the reasoning behind the name seems rather clear. For PyTorch, however, I cannot seem to come across why it is so named. Of course, I understand the "Py-" prefix and also know that PyTorch is a successor in some sense of Torch. But I am still wondering: what is the original idea behind the "-Torch" part? Is it known what the origin of the name is?

Response: Here a short answer, formed as another question: Torch, SMORCH ??? PyTorch developed from Torch7. A precursor to the original Torch was a library called SVM-Torch, which was developed around 2001. The SVM stands for Support Vector Machines. SVM-Torch is a decomposition algorithm similar to SVM-Light, but adapted to regression problems, according to this paper. Also around this time, G.W.Flake described the sequential minimal optimization algorithm (SMO), which could be used to train SVMs on sparse data sets, and this was incorporated into NODElib. Interestingly, this was called the SMORCH algorithm. You can find out more about SMORCH in the NODElib docs Optimization of the SVMs is: performed by a variation of John Platt's sequential minimal optimization (SMO) algorithm. This version of SMO is generalized for regression, uses kernel caching, and incorporates several heuristics; for these reasons, we refer to the optimization algorithm as SMORCH. So SMORCH = Sequential Minimal Optimization Regression Caching Heuristics I can't answer definitively, but my thinking is "Torch" is a riff or evolution of "Light" from SVM-Light combined with a large helping of SMORCHiness. You'd need to check in with the authors of SVMTorch and SVM-Light to confirm that this is indeed what "sparked" the name. It is reasonable to assume that the "TO" of Torch stands for some other optimization, rather than SMO, such as Tensor Optimization, but I haven't found any direct reference... yet.


Question:
What is the class definition of nn.Linear in PyTorch? What is self.hidden in the following code?
```
import torch.nn as nn
import torch.nn.functional as F

class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(784, 256)
        self.output = nn.Linear(256, 10)

    def forward(self, x):
        x = F.sigmoid(self.hidden(x))
        x = F.softmax(self.output(x), dim=1)
        return x
```
self.hidden is nn.Linear and it can take a tensor x as argument.

Response:
What is the class definition of nn.Linear in pytorch? From documentation: CLASS torch.nn.Linear(in_features, out_features, bias=True) Applies a linear transformation to the incoming data: y = x*W^T + b Parameters: in_features – size of each input sample (i.e. size of x) out_features – size of each output sample (i.e. size of y) bias – If set to False, the layer will not learn an additive bias. Default: True Note that the weights W have shape (out_features, in_features) and biases b have shape (out_features). They are initialized randomly and can be changed later (e.g. during the training of a Neural Network they are updated by some optimization algorithm). In your Neural Network, the self.hidden = nn.Linear(784, 256) defines a hidden (meaning that it is in between of the input and output layers), fully connected linear layer, which takes input x of shape (batch_size, 784), where batch size is the number of inputs (each of size 784) which are passed to the network at once (as a single tensor), and transforms it by the linear equation y = x*W^T + b into a tensor y of shape (batch_size, 256). It is further transformed by the sigmoid function, x = F.sigmoid(self.hidden(x)) (which is not a part of the nn.Linear but an additional step). Let's see a concrete example:
```
import torch
import torch.nn as nn

x = torch.tensor([[1.0, -1.0],
                  [0.0,  1.0],
                  [0.0,  0.0]])

in_features = x.shape[1]  # = 2
out_features = 2

m = nn.Linear(in_features, out_features)
```
where x contains three inputs (i.e. the batch size is 3), x[0], x[1] and x[3], each of size 2, and the output is going to be of shape (batch size, out_features) = (3, 2). The values of the parameters (weights and biases) are:
```

```python
m.weight
#Output
#tensor([[-0.4500,  0.5856],
#        [-0.1807, -0.4963]])
```

```python
m.bias
#Output
#tensor([ 0.2223, -0.6114])
#```
#(because they were initialized randomly, most likely you will get different values from the above) The output is:
#```
```

```python
y = m(x)
#Output
#tensor([[-0.8133, -0.2959],
#        [ 0.8079, -1.1077],
#        [ 0.2223, -0.6114]])
#```
#and (behind the scenes) it is computed as:
#```
#y = x.matmul(m.weight.t()) + m.bias  # y = x*W^T + b
#```
#i.e.
#```
#y[i,j] == x[i,0] * m.weight[j,0] + x[i,1] * m.weight[j,1] + m.bias[j]
#```
#where i is in interval [0, batch_size) and j in [0, out_features).
```


Question:
Backward function in PyTorch I have some question about pytorch's backward function I don't think I'm getting the right output :
```
import numpy as np
import torch
from torch.autograd import Variable
a = Variable(torch.FloatTensor([[1,2,3],[4,5,6]]), requires_grad=True)
out = a * a
out.backward(a)
print(a.grad)
```
the output is
```
tensor([[ 2.,  8., 18.],
        [32., 50., 72.]])
```
maybe it's 2*a*a but i think the output suppose to be
```
tensor([[ 2.,  4., 6.],
        [8., 10., 12.]])
```
2*a. cause d(x^2)/dx=2x

Response:
Please read carefully the documentation on backward() to better understand it. By default, pytorch expects backward() to be called for the last output of the network - the loss function. The loss function always outputs a scalar and therefore, the gradients of the scalar loss w.r.t all other variables/parameters is well defined (using the chain rule). Thus, by default, backward() is called on a scalar tensor and expects no arguments. For example:
```py
a = torch.tensor([[1,2,3],[4,5,6]], dtype=torch.float, requires_grad=True)
for i in range(2):
  for j in range(3):
    out = a[i,j] * a[i,j]
    out.backward()
print(a.grad)
```
yields
```
tensor([[ 2.,  4.,  6.],
        [ 8., 10., 12.]])
```
As expected: d(a^2)/da = 2a. However, when you call backward on the 2-by-3 out tensor (no longer a scalar function) - what do you expects a.grad to be? You'll actually need a 2-by-3-by-2-by-3 output: d out[i,j] / d a[k,l](!) Pytorch does not support this non-scalar function derivatives. Instead, pytorch assumes out is only an intermediate tensor and somewhere "upstream" there is a scalar loss function, that through chain rule provides d loss/ d out[i,j]. This "upstream" gradient is of size 2-by-3 and this is actually the argument you provide backward in this case: out.backward(g) where g_ij = d loss/ d out_ij. The gradients are then calculated by chain rule d loss / d a[i,j] = (d loss/d out[i,j]) * (d out[i,j] / d a[i,j]) Since you provided a as the "upstream" gradients you got
```
a.grad[i,j] = 2 * a[i,j] * a[i,j]
```
If you were to provide the "upstream" gradients to be all ones
```py
out.backward(torch.ones(2,3))
print(a.grad)
```
yields
```
tensor([[ 2.,  4.,  6.],
        [ 8., 10., 12.]])
```
As expected. It's all in the chain rule.


Question:
Calculate the accuracy every epoch in PyTorch I am working on a Neural Network problem, to classify data as 1 or 0. I am using Binary cross entropy loss to do this. The loss is fine, however, the accuracy is very low and isn't improving. I am assuming I did a mistake in the accuracy calculation. After every epoch, I am calculating the correct predictions after thresholding the output, and dividing that number by the total number of the dataset. Is there any thing wrong I did in the accuracy calculation? And why isn't it improving, but getting more worse? This is my code:
```py
net = Model()
criterion = torch.nn.BCELoss(size_average=True)
optimizer = torch.optim.SGD(net.parameters(), lr=0.1)

num_epochs = 100
for epoch in range(num_epochs):
    for i, (inputs,labels) in enumerate (train_loader):
        inputs = Variable(inputs.float())
        labels = Variable(labels.float())
        output = net(inputs)
        optimizer.zero_grad()
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

    #Accuracy
    output = (output>0.5).float()
    correct = (output == labels).float().sum()
    print("Epoch {}/{}, Loss: {:.3f}, Accuracy: {:.3f}".format(epoch+1,num_epochs, loss.data[0], correct/x.shape[0]))
```
And this is the strange output I get:
```
Epoch 1/100, Loss: 0.389, Accuracy: 0.035
Epoch 2/100, Loss: 0.370, Accuracy: 0.036
Epoch 3/100, Loss: 0.514, Accuracy: 0.030
Epoch 4/100, Loss: 0.539, Accuracy: 0.030
Epoch 5/100, Loss: 0.583, Accuracy: 0.029
Epoch 6/100, Loss: 0.439, Accuracy: 0.031
Epoch 7/100, Loss: 0.429, Accuracy: 0.034
Epoch 8/100, Loss: 0.408, Accuracy: 0.035
Epoch 9/100, Loss: 0.316, Accuracy: 0.035
Epoch 10/100, Loss: 0.436, Accuracy: 0.035
Epoch 11/100, Loss: 0.365, Accuracy: 0.034
Epoch 12/100, Loss: 0.485, Accuracy: 0.031
Epoch 13/100, Loss: 0.392, Accuracy: 0.033
Epoch 14/100, Loss: 0.494, Accuracy: 0.030
Epoch 15/100, Loss: 0.369, Accuracy: 0.035
Epoch 16/100, Loss: 0.495, Accuracy: 0.029
Epoch 17/100, Loss: 0.415, Accuracy: 0.034
Epoch 18/100, Loss: 0.410, Accuracy: 0.035
Epoch 19/100, Loss: 0.282, Accuracy: 0.038
Epoch 20/100, Loss: 0.499, Accuracy: 0.031
Epoch 21/100, Loss: 0.446, Accuracy: 0.030
Epoch 22/100, Loss: 0.585, Accuracy: 0.026
Epoch 23/100, Loss: 0.419, Accuracy: 0.035
Epoch 24/100, Loss: 0.492, Accuracy: 0.031
Epoch 25/100, Loss: 0.537, Accuracy: 0.031
Epoch 26/100, Loss: 0.439, Accuracy: 0.033
Epoch 27/100, Loss: 0.421, Accuracy: 0.035
Epoch 28/100, Loss: 0.532, Accuracy: 0.034
Epoch 29/100, Loss: 0.234, Accuracy: 0.038
Epoch 30/100, Loss: 0.492, Accuracy: 0.027
Epoch 31/100, Loss: 0.407, Accuracy: 0.035
Epoch 32/100, Loss: 0.305, Accuracy: 0.038
Epoch 33/100, Loss: 0.663, Accuracy: 0.025
Epoch 34/100, Loss: 0.588, Accuracy: 0.031
Epoch 35/100, Loss: 0.329, Accuracy: 0.035
Epoch 36/100, Loss: 0.474, Accuracy: 0.033
Epoch 37/100, Loss: 0.535, Accuracy: 0.031
Epoch 38/100, Loss: 0.406, Accuracy: 0.033
Epoch 39/100, Loss: 0.513, Accuracy: 0.030
Epoch 40/100, Loss: 0.593, Accuracy: 0.030
Epoch 41/100, Loss: 0.265, Accuracy: 0.036
Epoch 42/100, Loss: 0.576, Accuracy: 0.031
Epoch 43/100, Loss: 0.565, Accuracy: 0.027
Epoch 44/100, Loss: 0.576, Accuracy: 0.030
Epoch 45/100, Loss: 0.396, Accuracy: 0.035
Epoch 46/100, Loss: 0.423, Accuracy: 0.034
Epoch 47/100, Loss: 0.489, Accuracy: 0.033
Epoch 48/100, Loss: 0.591, Accuracy: 0.029
Epoch 49/100, Loss: 0.415, Accuracy: 0.034
Epoch 50/100, Loss: 0.291, Accuracy: 0.039
Epoch 51/100, Loss: 0.395, Accuracy: 0.033
Epoch 52/100, Loss: 0.540, Accuracy: 0.026
Epoch 53/100, Loss: 0.436, Accuracy: 0.033
Epoch 54/100, Loss: 0.346, Accuracy: 0.036
Epoch 55/100, Loss: 0.519, Accuracy: 0.029
Epoch 56/100, Loss: 0.456, Accuracy: 0.031
Epoch 57/100, Loss: 0.425, Accuracy: 0.035
Epoch 58/100, Loss: 0.311, Accuracy: 0.039
Epoch 59/100, Loss: 0.406, Accuracy: 0.034
Epoch 60/100, Loss: 0.360, Accuracy: 0.035
Epoch 61/100, Loss: 0.476, Accuracy: 0.030
Epoch 62/100, Loss: 0.404, Accuracy: 0.034
Epoch 63/100, Loss: 0.382, Accuracy: 0.036
Epoch 64/100, Loss: 0.538, Accuracy: 0.031
Epoch 65/100, Loss: 0.392, Accuracy: 0.034
Epoch 66/100, Loss: 0.434, Accuracy: 0.033
Epoch 67/100, Loss: 0.479, Accuracy: 0.031
Epoch 68/100, Loss: 0.494, Accuracy: 0.031
Epoch 69/100, Loss: 0.415, Accuracy: 0.034
Epoch 70/100, Loss: 0.390, Accuracy: 0.036
Epoch 71/100, Loss: 0.330, Accuracy: 0.038
Epoch 72/100, Loss: 0.449, Accuracy: 0.030
Epoch 73/100, Loss: 0.315, Accuracy: 0.039
Epoch 74/100, Loss: 0.450, Accuracy: 0.031
Epoch 75/100, Loss: 0.562, Accuracy: 0.030
Epoch 76/100, Loss: 0.447, Accuracy: 0.031
Epoch 77/100, Loss: 0.408, Accuracy: 0.038
Epoch 78/100, Loss: 0.359, Accuracy: 0.034
Epoch 79/100, Loss: 0.372, Accuracy: 0.035
Epoch 80/100, Loss: 0.452, Accuracy: 0.034
Epoch 81/100, Loss: 0.360, Accuracy: 0.035
Epoch 82/100, Loss: 0.453, Accuracy: 0.031
Epoch 83/100, Loss: 0.578, Accuracy: 0.030
Epoch 84/100, Loss: 0.537, Accuracy: 0.030
Epoch 85/100, Loss: 0.483, Accuracy: 0.035
Epoch 86/100, Loss: 0.343, Accuracy: 0.036
Epoch 87/100, Loss: 0.439, Accuracy: 0.034
Epoch 88/100, Loss: 0.686, Accuracy: 0.023
Epoch 89/100, Loss: 0.265, Accuracy: 0.039
Epoch 90/100, Loss: 0.369, Accuracy: 0.035
Epoch 91/100, Loss: 0.521, Accuracy: 0.027
Epoch 92/100, Loss: 0.662, Accuracy: 0.027
Epoch 93/100, Loss: 0.581, Accuracy: 0.029
Epoch 94/100, Loss: 0.322, Accuracy: 0.034
Epoch 95/100, Loss: 0.375, Accuracy: 0.035
Epoch 96/100, Loss: 0.575, Accuracy: 0.031
Epoch 97/100, Loss: 0.489, Accuracy: 0.030
Epoch 98/100, Loss: 0.435, Accuracy: 0.033
Epoch 99/100, Loss: 0.440, Accuracy: 0.031
Epoch 100/100, Loss: 0.444, Accuracy: 0.033
```

Response:
A better way would be calculating correct right after optimization step
```py
for epoch in range(num_epochs):

    correct = 0
    for i, (inputs,labels) in enumerate (train_loader):
        ...
        output = net(inputs)
        ...
        optimizer.step()

        correct += (output == labels).float().sum()

    accuracy = 100 * correct / len(trainset)
    # trainset, not train_loader
    # probably x in your case

    print("Accuracy = {}".format(accuracy))
```


Question:
pytorch: "multi-target not supported" error message So I want to classify some (3, 50, 50) pictures. First I loaded the dataset from the file without a dataloader or batches, it worked. Now, after adding both things I get that error:
```
RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15
```
I found a lot of answers in the internet, mostly to use target.squeeze(1) but it doesn´t work for me. My target-batch looks like following:
```
tensor([[1, 0],
        [1, 0],
        [1, 0],
        [1, 0],
        [1, 0],
        [1, 0],
        [1, 0],
        [1, 0]], device='cuda:0')
```
Shouldn't that be okay? Here the full code (notice that Im only creating the structure of the model on which Im going to apply the full and correct dataset afterwards, because I dont have the full data yet, only 32 pictures and no labels, thats why I added torch.tensor([1, 0]) as a placeholder for all labels):
```py
import torch
import torch.utils.data
import torch.nn as nn
import torch.nn.functional as F
import torch.optim
from torch.autograd import Variable

import numpy as np
from PIL import Image

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()

        # model structur:
        self.conv1 = nn.Conv2d(3, 10, kernel_size=(5,5),  stride=(1,1))
        self.conv2 = nn.Conv2d(10, 20, kernel_size=(5,5),  stride=(1,1))            # with mapool: output = 20 * (9,9) feature-maps -> flatten
        self.fc1 = nn.Linear(20*9*9, 250)
        self.fc2 = nn.Linear(250, 100)
        self.fc3 = nn.Linear(100, 2)

    def forward(self, x):
        # conv layers
        x = F.relu(self.conv1(x))   # shape: 1, 10, 46, 46
        x = F.max_pool2d(x, 2, 2)   # shape: 1, 10, 23, 23
        x = F.relu(self.conv2(x))   # shape: 1, 20, 19, 19
        x = F.max_pool2d(x, 2, 2)   # shape: 1, 20, 9, 9

        # flatten to dense layer:
        x = x.view(-1, 20*9*9)

        # dense layers
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        output = F.log_softmax(self.fc3(x), dim=1)
        return output

class Run:
    def __init__(self, epochs, learning_rate, dropout, momentum):
        # load model
        self.model = Model().cuda()

        # hyperparameters:
        self.epochs = epochs
        self.learning_rate = learning_rate
        self.dropout = dropout

    def preporcessing(self):
        dataset_folder = "/media/theodor/hdd/Programming/BWKI/dataset/bilder/"

        dataset = []
        for i in range(0, 35):
            sample_image = Image.open(dataset_folder + str(i) + ".png")
            data = torch.from_numpy(np.array(sample_image)).type("torch.Tensor").reshape(3, 50, 50)
            target = torch.tensor([[1, 0]])
            sample = (data, target)
            dataset.append(sample)

        train_loader = torch.utils.data.DataLoader(dataset, batch_size=8)

        return train_loader

    def train(self):
        train_set = self.preporcessing()

        criterion = nn.CrossEntropyLoss()
        optimizer = torch.optim.SGD(self.model.parameters(), lr=self.learning_rate)
        for epoch in range(self.epochs):
            epoch_loss = 0
            for i, data in enumerate(train_set, 0):

                sample, target = data
                # set data as cuda varibale
                sample = Variable(sample.float().cuda())
                target = Variable(target.cuda())
                # initialize optimizer
                optimizer.zero_grad()
                # predict
                output = self.model(sample)
                # backpropagation
                print(output, target.squeeze(1))
                loss = criterion(output, target.squeeze(1))    # ERROR MESSAGE: RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15
                loss.backward()
                optimizer.step()

                epoch_loss += loss.item()

            print("loss after epoch [", epoch, "|", self.epochs, "] :", epoch_loss)

run = Run(10, 0.001, 0.5, 0.9)
run.train()
```
So I expected it to start training (of course not learning anything because the labels are wrong).

Response: For nn.CrossEntropyLoss the target has to be a single number from the interval [0, #classes] instead of a one-hot encoded target vector. Your target is [1, 0], thus PyTorch thinks you want to have multiple labels per input which is not supported. Replace your one-hot-encoded targets: [1, 0] --> 0 [0, 1] --> 1


Question: What is the difference between .flatten() and .view(-1) in PyTorch? Both .flatten() and .view(-1) flatten a tensor in PyTorch. What's the difference? Does .flatten() copy the data of the tensor? Is .view(-1) faster? Is there any situation that .flatten() doesn't work?

Response: In addition to @adeelh's comment, there is another difference: torch.flatten() results in a .reshape(), and the differences between .reshape() and .view() are: [...] torch.reshape may return a copy or a view of the original tensor. You can not count on that to return a view or a copy. Another difference is that reshape() can operate on both contiguous and non-contiguous tensor while view() can only operate on contiguous tensor. Also see here about the meaning of contiguous For context: The community requested for a flatten function for a while, and after Issue #7743, the feature was implemented in the PR #8578. You can see the implementation of flatten here, where a call to .reshape() can be seen in return line.


Question:
Load data into GPU directly using PyTorch In training loop, I load a batch of data into CPU and then transfer it to GPU:
```
import torch.utils as utils

train_loader = utils.data.DataLoader(train_dataset, batch_size=128, shuffle=True, num_workers=4, pin_memory=True)

for inputs, labels in train_loader:
    inputs, labels = inputs.to(device), labels.to(device)
```
This way of loading data is very time-consuming. Any way to directly load data into GPU without transfer step ?

Response:
@PeterJulian first of all thanks for the reply. As far as I know there is no single line command for loading a whole dataset to GPU. Actually in my reply I meant to use .to(device) in the __init__ of the data loader. There are some examples in the link that I had shared previously. Also, I left an example data loader code below. Hope both the examples in the link and the code below helps.
```
class SampleDataset(Dataset):
    def __init__(self, device='cuda'):
        super(SampleDataset, self).__init__()
        self.data = torch.ones(1000)
        self.data = self.data.to(device)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, i):
        element = self.data[i]
        return element
```


Question:
How do I write a PyTorch sequential model? How do I write a sequential model in PyTorch, just like what we can do with Keras? I tried:
```
import torch
import torch.nn as nn

net = nn.Sequential()
net.add(nn.Linear(3, 4))
net.add(nn.Sigmoid())
net.add(nn.Linear(4, 1))
net.add(nn.Sigmoid())
net.float()
```
But I get the error: AttributeError: 'Sequential' object has no attribute 'add'

Response:
Sequential does not have an add method at the moment, though there is some debate about adding this functionality. As you can read in the documentation nn.Sequential takes as argument the layers separated as a sequence of arguments or an OrderedDict. If you have a model with lots of layers, you can create a list first and then use the * operator to expand the list into positional arguments, like this:
```py
layers = []
layers.append(nn.Linear(3, 4))
layers.append(nn.Sigmoid())
layers.append(nn.Linear(4, 1))
layers.append(nn.Sigmoid())

net = nn.Sequential(*layers)
```
This will result in a similar structure of your code, as adding directly.


Question:
How to compute the cosine_similarity in pytorch for all rows in a matrix with respect to all rows in another matrix In pytorch, given that I have 2 matrixes how would I compute cosine similarity of all rows in each with all rows in the other. For example Given the input =
```
matrix_1 = [a b]
           [c d]
matrix_2 = [e f]
           [g h]
```
I would like the output to be output =
```
[cosine_sim([a b] [e f])  cosine_sim([a b] [g h])]
 [cosine_sim([c d] [e f])  cosine_sim([c d] [g h])]
```
At the moment I am using torch.nn.functional.cosine_similarity(matrix_1, matrix_2) which returns the cosine of the row with only that corresponding row in the other matrix. In my example I have only 2 rows, but I would like a solution which works for many rows. I would even like to handle the case where the number of rows in the each matrix is different. I realize that I could use the expand, however I want to do it without using such a large memory footprint.

Response:
By manually computing the similarity and playing with matrix multiplication + transposition:
```python
import torch
from scipy import spatial
import numpy as np

a = torch.randn(2, 2)
b = torch.randn(3, 2) # different row number, for the fun

# Given that cos_sim(u, v) = dot(u, v) / (norm(u) * norm(v))
#                          = dot(u / norm(u), v / norm(v))
# We fist normalize the rows, before computing their dot products via transposition:
a_norm = a / a.norm(dim=1)[:, None]
b_norm = b / b.norm(dim=1)[:, None]
res = torch.mm(a_norm, b_norm.transpose(0,1))
print(res)
#  0.9978 -0.9986 -0.9985
# -0.8629  0.9172  0.9172

# -------
# Let's verify with numpy/scipy if our computations are correct:
a_n = a.numpy()
b_n = b.numpy()
res_n = np.zeros((2, 3))
for i in range(2):
    for j in range(3):
        # cos_sim(u, v) = 1 - cos_dist(u, v)
        res_n[i, j] = 1 - spatial.distance.cosine(a_n[i], b_n[j])
print(res_n)
# [[ 0.9978022  -0.99855876 -0.99854881]
#  [-0.86285472  0.91716063  0.9172349 ]]
```


Question: set `torch.backends.cudnn.benchmark = True` or not? I am using pytorch and I wonder if I should use torch.backends.cudnn.benchmark = True. I find on google that I should use it when computation graph does not change. What is computation graph in pytorch?

Response: If your model does not change and your input sizes remain the same - then you may benefit from setting torch.backends.cudnn.benchmark = True. However, if your model changes: for instance, if you have layers that are only "activated" when certain conditions are met, or you have layers inside a loop that can be iterated a different number of times, then setting torch.backends.cudnn.benchmark = True might stall your execution.


Question:
PyTorch set_grad_enabled(False) vs with no_grad(): Assuming autograd is on (as it is by default), is there any difference (besides indent) between doing:
```
with torch.no_grad():
    <code>
```
and
```
torch.set_grad_enabled(False)
<code>
torch.set_grad_enabled(True)
```

Response:
Actually no, there no difference in the way used in the question. When you take a look at the source code of no_grad. You see that it is actually using torch.set_grad_enabled to archive this behaviour:
```py
class no_grad(object):
    r"""Context-manager that disabled gradient calculation.

    Disabling gradient calculation is useful for inference, when you are sure
    that you will not call :meth:`Tensor.backward()`. It will reduce memory
    consumption for computations that would otherwise have `requires_grad=True`.
    In this mode, the result of every computation will have
    `requires_grad=False`, even when the inputs have `requires_grad=True`.

    Also functions as a decorator.

    Example::

        >>> x = torch.tensor([1], requires_grad=True)
        >>> with torch.no_grad():
        ...   y = x * 2
        >>> y.requires_grad
        False
        >>> @torch.no_grad()
        ... def doubler(x):
        ...     return x * 2
        >>> z = doubler(x)
        >>> z.requires_grad
        False
    """

    def __init__(self):
        self.prev = torch.is_grad_enabled()

    def __enter__(self):
        torch._C.set_grad_enabled(False)

    def __exit__(self, *args):
        torch.set_grad_enabled(self.prev)
        return False

    def __call__(self, func):
        @functools.wraps(func)
        def decorate_no_grad(*args, **kwargs):
            with self:
                return func(*args, **kwargs)
        return decorate_no_grad
```
However there is an additional functionality of torch.set_grad_enabled over torch.no_grad when used in a with-statement which lets you control to switch on or off gradient computation:
```py

```python
x = torch.tensor([1], requires_grad=True)
#Output
#    >>> is_train = False
#    >>> with torch.set_grad_enabled(is_train):
#    ...   y = x * 2
#    >>> y.requires_grad
#```
#https://pytorch.org/docs/stable/_modules/torch/autograd/grad_mode.html Edit: @TomHale Regarding your comment. I just made a short test with PyTorch 1.0 and it turned out that the gradient will be active:
#```
#import torch
#w = torch.rand(5, requires_grad=True)
#print('Grad Before:', w.grad)
#torch.set_grad_enabled(False)
#with torch.enable_grad():
#    scalar = w.sum()
#    scalar.backward()
#    # Gradient tracking will be enabled here.
#torch.set_grad_enabled(True)
```

print('Grad After:', w.grad)
```
Output:
```
Grad Before: None
Grad After: tensor([1., 1., 1., 1., 1.])
```
So gradients will be computed in this setting. The other setting you posted in your answer also yields to the same result:
```
import torch
w = torch.rand(5, requires_grad=True)
print('Grad Before:', w.grad)
with torch.no_grad():
    with torch.enable_grad():
        # Gradient tracking IS enabled here.
        scalar = w.sum()
        scalar.backward()

print('Grad After:', w.grad)
```
Output:
```
Grad Before: None
Grad After: tensor([1., 1., 1., 1., 1.])
```


Question: What "exactly" happens inside embedding layer in pytorch? From multiple searches and pytorch documentation itself I could figure out that inside embedding layer there is a lookup table where the embedding vectors are stored. What I am not able to understand: what exactly happens during training in this layer? What are the weights and how the gradients of those weights are computed? My intuition is that at least there should be a function with some parameters that produces the keys for the lookup table. If so, then what is that function? Any help in this will be appreciated. Thanks.

Response:
That is a really good question! The embedding layer of PyTorch (same goes for Tensorflow) serves as a lookup table just to retrieve the embeddings for each of the inputs, which are indices. Consider the following case, you have a sentence where each word is tokenized. Therefore, each word in your sentence is represented with a unique integer (index). In case the list of indices (words) is [1, 5, 9], and you want to encode each of the words with a 50 dimensional vector (embedding), you can do the following:
```py
# The list of tokens
tokens = torch.tensor([0,5,9], dtype=torch.long)
# Define an embedding layer, where you know upfront that in total you
# have 10 distinct words, and you want each word to be encoded with
# a 50 dimensional vector
embedding = torch.nn.Embedding(num_embeddings=10, embedding_dim=50)
# Obtain the embeddings for each of the words in the sentence
embedded_words = embedding(tokens)
```
Now, to answer your questions: During the forward pass, the values for each of the tokens in your sentence are going to be obtained in a similar way as the Numpy's indexing works. Because in the backend, this is a differentiable operation, during the backward pass (training), Pytorch is going to compute the gradients for each of the embeddings and readjust them accordingly. The weights are the embeddings themselves. The word embedding matrix is actually a weight matrix that will be learned during training. There is no actual function per se. As we defined above, the sentence is already tokenized (each word is represented with a unique integer), and we can just obtain the embeddings for each of the tokens in the sentence. Finally, as I mentioned the example with the indexing many times, let us try it out.
```py
# Let us assume that we have a pre-trained embedding matrix
pretrained_embeddings = torch.rand(10, 50)
# We can initialize our embedding module from the embedding matrix
embedding = torch.nn.Embedding.from_pretrained(pretrained_embeddings)
# Some tokens
tokens = torch.tensor([1,5,9], dtype=torch.long)

# Token embeddings from the lookup table
lookup_embeddings = embedding(tokens)
# Token embeddings obtained with indexing
indexing_embeddings = pretrained_embeddings[tokens]
# Voila! They are the same
np.testing.assert_array_equal(lookup_embeddings.numpy(), indexing_embeddings.numpy())
```


Question:
PyTorch : How to properly create a list of nn.Linear() I have created a class that has nn.Module as subclass. In my class, I have to create N number of linear transformation, where N is given as class parameters. I therefore proceed as follow :
```
self.list_1 = []

    for i in range(N):
        self.list_1.append(nn.Linear(self.x, 1, bias=mlp_bias))
```
In the forward method, i call these matrices (with list_1[i]) and concat the results. Two things : 1) Even though I use model.cuda(), these Linear transform are used on cpu and i get the following error : RuntimeError: Expected object of type Variable[torch.cuda.FloatTensor] but found type Variable[torch.FloatTensor] for argument #1 'mat2' I have to do
```
self.list_1.append(nn.Linear(self.x, 1, bias=mlp_bias).cuda())
```
This is not required if instead, i do :
```
self.nn = nn.Linear(self.x, 1, bias=mlp_bias)
```
and then use self.nn directly. 2) For more obvious reason, when I print(model) in my main, the Linear matrices in my list arent printed. Is there any other way. maybe using bmm ? I find it less easy, and i actually want to have my N results separately. Thank you in advance, M

Response:
You can use nn.ModuleList to wrap your list of linear layers as explained here
```
self.list_1 = nn.ModuleList(self.list_1)
```


Question:
Pytorch Change the learning rate based on number of epochs When I set the learning rate and find the accuracy cannot increase after training few epochs
```
optimizer = optim.Adam(model.parameters(), lr = 1e-4)

n_epochs = 10
for i in range(n_epochs):

    // some training here
```
If I want to use a step decay: reduce the learning rate by a factor of 10 every 5 epochs, how can I do so?

Response:
You can use learning rate scheduler torch.optim.lr_scheduler.StepLR
```py
import torch.optim.lr_scheduler.StepLR
scheduler = StepLR(optimizer, step_size=5, gamma=0.1)
```
Decays the learning rate of each parameter group by gamma every step_size epochs see docs here Example from docs
```py
# Assuming optimizer uses lr = 0.05 for all groups
# lr = 0.05     if epoch < 30
# lr = 0.005    if 30 <= epoch < 60
# lr = 0.0005   if 60 <= epoch < 90
# ...
scheduler = StepLR(optimizer, step_size=30, gamma=0.1)
for epoch in range(100):
    train(...)
    validate(...)
    scheduler.step()
```
Example:
```py
import torch
import torch.optim as optim

optimizer = optim.SGD([torch.rand((2,2), requires_grad=True)], lr=0.1)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)
```
```py
for epoch in range(1, 21):
    scheduler.step()
    print('Epoch-{0} lr: {1}'.format(epoch, optimizer.param_groups[0]['lr']))
    if epoch % 5 == 0:print()
```
```py
Epoch-1 lr: 0.1
Epoch-2 lr: 0.1
Epoch-3 lr: 0.1
Epoch-4 lr: 0.1
Epoch-5 lr: 0.1

Epoch-6 lr: 0.010000000000000002
Epoch-7 lr: 0.010000000000000002
Epoch-8 lr: 0.010000000000000002
Epoch-9 lr: 0.010000000000000002
Epoch-10 lr: 0.010000000000000002

Epoch-11 lr: 0.0010000000000000002
Epoch-12 lr: 0.0010000000000000002
Epoch-13 lr: 0.0010000000000000002
Epoch-14 lr: 0.0010000000000000002
Epoch-15 lr: 0.0010000000000000002

Epoch-16 lr: 0.00010000000000000003
Epoch-17 lr: 0.00010000000000000003
Epoch-18 lr: 0.00010000000000000003
Epoch-19 lr: 0.00010000000000000003
Epoch-20 lr: 0.00010000000000000003
```
More on How to adjust Learning Rate - torch.optim.lr_scheduler provides several methods to adjust the learning rate based on the number of epochs.


Question:
PyTorch model input shape I loaded a custom PyTorch model and I want to find out its input shape. Something like this:
```
model.input_shape
```
Is it possible to get this information? Update: print() and summary() don't show this model's input shape, so they are not what I'm looking for.

Response:
PyTorch flexibility PyTorch models are very flexible objects, to the point where they do not enforce or generally expect a fixed input shape for data. If you have certain layers there may be constraints e.g: a flatten followed by a fully connected layer of width N would enforce the dimensions of your original input (M1 x M2 x ... Mn) to have a product equal to N a 2d convolution of N input channels would enforce the data to be 3 dimensionsal, with the first dimension having size N But as you can see neither of these enforce the total shape of the data. We might not realize it right now, but in more complex models, getting the size of the first linear layer right is sometimes a source of frustration. We’ve heard stories of famous practitioners putting in arbitrary numbers and then relying on error messages from PyTorch to backtrack the correct sizes for their linear layers. Lame, eh? Nah, it’s all legit! Deep Learning with PyTorch Investigation Simple case: First layer is Fully Connected If your model's first layer is a fully connected one, then the first layer in print(model) will detail the expected dimensionality of a single sample. Ambiguous case: CNN If it is a convolutional layer however, since these are dynamic and will stride as long/wide as the input permits, there is no simple way to retrieve this info from the model itself.1 This flexibility means that for many architectures multiple compatible input sizes2 will all be acceptable by the network. This is a feature of PyTorch's Dynamic computational graph. Manual inspection What you will need to do is investigate the network architecture, and once you've found an interpretable layer (if one is present e.g. fully connected) "work backwards" with its dimensions, determining how the previous layers (e.g. poolings and convolutions) have compressed/modified it. Example e.g. in the following model from Deep Learning with PyTorch (8.5.1):
```py
class NetWidth(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(32, 16, kernel_size=3, padding=1)
        self.fc1 = nn.Linear(16 * 8 * 8, 32)
        self.fc2 = nn.Linear(32, 2)

    def forward(self, x):
        out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)
        out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)
        out = out.view(-1, 16 * 8 * 8)
        out = torch.tanh(self.fc1(out))
        out = self.fc2(out)
        return out
```
We see the model takes an input 2.d. image with 3 channels and: Conv2d -> sends it to an image of the same size with 32 channels max_pool2d(,2) -> halves the size of the image in each dimension Conv2d -> sends it to an image of the same size with 16 channels max_pool2d(,2) -> halves the size of the image in each dimension view -> reshapes the image Linear -> takes a tensor of size 16 * 8 * 8 and sends to size 32 ... So working backwards, we have: a tensor of shape 16 * 8 * 8 un-reshaped into shape (channels x height x width) un-max_pooled in 2d with factor 2, so height and width un-halved un-convolved from 16 channels to 32 Hypothesis: It is likely 16 in the product thus refers to the number of channels, and that the image seen by view was of shape (channels, 8,8), and currently is (channels, 16,16)2 un-max_pooled in 2d with factor 2, so height and width un-halved again (channels, 32,32) un-convolved from 32 channels to 3 So assuming the kernel_size and padding are sufficient that the convolutions themselves maintain image dimensions, it is likely that the input image is of shape (3,32,32) i.e. RGB 32x32 pixel square images. Notes: Even the external package pytorch-summary requires you provide the input shape in order to display the shape of the output of each layer. It could however be any 2 numbers whose produce equals 8*8 e.g. (64,1), (32,2), (16,4) etc however since the code is written as 8*8 it is likely the authors used the actual dimensions.


Question:
Difference between 1 LSTM with num_layers = 2 and 2 LSTMs in pytorch I am new to deep learning and currently working on using LSTMs for language modeling. I was looking at the pytorch documentation and was confused by it. If I create a
```
nn.LSTM(input_size, hidden_size, num_layers)
```
where hidden_size = 4 and num_layers = 2, I think I will have an architecture something like:
```
op0    op1 ....
LSTM -> LSTM -> h3
LSTM -> LSTM -> h2
LSTM -> LSTM -> h1
LSTM -> LSTM -> h0
x0     x1 .....
```
If I do something like
```
nn.LSTM(input_size, hidden_size, 1)
nn.LSTM(input_size, hidden_size, 1)
```
I think the network architecture will look exactly like above. Am I wrong? And if yes, what is the difference between these two?

Response:
The multi-layer LSTM is better known as stacked LSTM where multiple layers of LSTM are stacked on top of each other. Your understanding is correct. The following two definitions of stacked LSTM are same.
```
nn.LSTM(input_size, hidden_size, 2)
```
and
```
nn.Sequential(OrderedDict([
    ('LSTM1', nn.LSTM(input_size, hidden_size, 1),
    ('LSTM2', nn.LSTM(hidden_size, hidden_size, 1)
]))
```
Here, the input is feed into the lowest layer of LSTM and then the output of the lowest layer is forwarded to the next layer and so on so forth. Please note, the output size of the lowest LSTM layer and the rest of the LSTM layer's input size is hidden_size. However, you may have seen people defined stacked LSTM in the following way:
```
rnns = nn.ModuleList()
for i in range(nlayers):
    input_size = input_size if i == 0 else hidden_size
    rnns.append(nn.LSTM(input_size, hidden_size, 1))
```
The reason people sometimes use the above approach is that if you create a stacked LSTM using the first two approaches, you can't get the hidden states of each individual layer. Check out what LSTM returns in PyTorch. So, if you want to have the intermedia layer's hidden states, you have to declare each individual LSTM layer as a single LSTM and run through a loop to mimic the multi-layer LSTM operations. For example:
```
outputs = []
for i in range(nlayers):
    if i != 0:
        sent_variable = F.dropout(sent_variable, p=0.2, training=True)
    output, hidden = rnns[i](sent_variable)
    outputs.append(output)
    sent_variable = output
```
In the end, outputs will contain all the hidden states of each individual LSTM layer.


Question: How to convert a pytorch tensor into a numpy array? How do I convert a torch tensor to numpy?

Response:
copied from pytorch doc:
```
a = torch.ones(5)
print(a)
```
tensor([1., 1., 1., 1., 1.])
```
b = a.numpy()
print(b)
```
[1. 1. 1. 1. 1.] Following from the below discussion with @John: In case the tensor is (or can be) on GPU, or in case it (or it can) require grad, one can use
```
t.detach().cpu().numpy()
```
I recommend to uglify your code only as much as required.


Question: How to check if a tensor is on cuda or send it to cuda in Pytorch? I have a tensor t = torch.zeros((4, 5, 6)) How to check if it is on gpu or not, and send it to gpu and back?

Response:
From the pytorch forum use t.is_cuda, t.cuda(), t.cpu()
```
t = torch.randn(2,2)
t.is_cuda  # returns False
t = torch.randn(2,2).cuda()
t.is_cuda  # returns True
t = t.cpu()
t.is_cuda  # returns False
```
When passing to and from gpu and cpu, new arrays are allocated on the relevant device.


Question:
Pytorch matmul - RuntimeError: "addmm_impl_cpu_" not implemented for 'Half' When I am running pytorch matmul, the following error is thrown:
```
Traceback (most recent call last):
  File "/home/omrastogi/Desktop/Side/One-Class-Classification-Customer-Complaints/pattern.py", line 71, in <module>
    print(obj.infer(list([df.text[0]]), list([df.reason[0]])))
  File "/home/omrastogi/Desktop/Side/One-Class-Classification-Customer-Complaints/pattern.py", line 45, in infer
    cos_sm = self.batch_cosine_similarity(enc1, enc2)
  File "/home/omrastogi/Desktop/Side/One-Class-Classification-Customer-Complaints/pattern.py", line 51, in batch_cosine_similarity
    dot_prd = torch.matmul(inp1, inp2.transpose(0, 1))
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
```
inp1 --> [1256] inp2 --> [1256]

Response: The error was throwing because the data type of operands was float16. Changing it back to float32 solved the problem. I guess float16 is for GPU implementation only.


Question: PyTorch LogSoftmax vs Softmax for CrossEntropyLoss I understand that PyTorch's LogSoftmax function is basically just a more numerically stable way to compute Log(Softmax(x)). Softmax lets you convert the output from a Linear layer into a categorical probability distribution. The pytorch documentation says that CrossEntropyLoss combines nn.LogSoftmax() and nn.NLLLoss() in one single class. Looking at NLLLoss, I'm still confused...Are there 2 logs being used? I think of negative log as information content of an event. (As in entropy) After a bit more looking, I think that NLLLoss assumes that you're actually passing in log probabilities instead of just probabilities. Is this correct? It's kind of weird if so...

Response: Yes, NLLLoss takes log-probabilities (log(softmax(x))) as input. Why?. Because if you add a nn.LogSoftmax (or F.log_softmax) as the final layer of your model's output, you can easily get the probabilities using torch.exp(output), and in order to get cross-entropy loss, you can directly use nn.NLLLoss. Of course, log-softmax is more stable as you said. And, there is only one log (it's in nn.LogSoftmax). There is no log in nn.NLLLoss. nn.CrossEntropyLoss() combines nn.LogSoftmax() (that is, log(softmax(x))) and nn.NLLLoss() in one single class. Therefore, the output from the network that is passed into nn.CrossEntropyLoss needs to be the raw output of the network (called logits), not the output of the softmax function.


Question:
What is the difference between view() and unsqueeze() in Torch? Use of unsqueeze():
```
input = torch.Tensor(2, 4, 3) # input: 2 x 4 x 3
print(input.unsqueeze(0).size()) # prints - torch.size([1, 2, 4, 3])
```
Use of view():
```
input = torch.Tensor(2, 4, 3) # input: 2 x 4 x 3
print(input.view(1, -1, -1, -1).size()) # prints - torch.size([1, 2, 4, 3])
```
According to documentation, unsqueeze() inserts singleton dim at position given as parameter and view() creates a view with different dimensions of the storage associated with tensor. What view() does is clear to me, but I am unable to distinguish it from unsqueeze(). Moreover, I don't understand when to use view() and when to use unsqueeze()? Any help with good explanation would be appreciated!

Response: view() can only take a single -1 argument. So, if you want to add a singleton dimension, you would need to provide all the dimensions as arguments. For e.g., if A is a 2x3x4 tensor, to add a singleton dimension, you would need to do A:view(2, 1, 3, 4). However, sometimes, the dimensionality of the input is unknown when the operation is being used. Thus, we dont know that A is 2x3x4, but we would still like to insert a singleton dimension. This happens a lot when using minibatches of tensors, where the last dimension is usually unknown. In these cases, the nn.Unsqueeze is useful and lets us insert the dimension without explicitly being aware of the other dimensions when writing the code.


Question:
Pytorch: Why is the memory occupied by the `tensor` variable so small? In Pytorch 1.0.0, I found that a tensor variable occupies very small memory. I wonder how it stores so much data. Here's the code.
```
a = np.random.randn(1, 1, 128, 256)
b = torch.tensor(a, device=torch.device('cpu'))

a_size = sys.getsizeof(a)
b_size = sys.getsizeof(b)
```
a_size is 262288. b_size is 72.

Response:
The answer is in two parts. From the documentation of sys.getsizeof, firstly All built-in objects will return correct results, but this does not have to hold true for third-party extensions as it is implementation specific. so it could be that for tensors __sizeof__ is undefined or defined differently than you would expect - this function is not something you can rely on. Secondly Only the memory consumption directly attributed to the object is accounted for, not the memory consumption of objects it refers to. which means that if the torch.Tensor object merely holds a reference to the actual memory, this won't show in sys.getsizeof. This is indeed the case, if you check the size of the underlying storage instead, you will see the expected number
```
import torch, sys
b = torch.randn(1, 1, 128, 256, dtype=torch.float64)
sys.getsizeof(b)
>> 72
sys.getsizeof(b.storage())
>> 262208
```
Note: I am setting dtype to float64 explicitly, because that is the default dtype in numpy, whereas torch uses float32 by default.


Question:
Pytorch tensor, how to switch channel position - Runtime error I have my training dataset as below, where X_train is 3D with 3 channels Shape of X_Train: (708, 256, 3) Shape of Y_Train: (708, 4) Then I convert them into a tensor and input into the dataloader:
```
X_train=torch.from_numpy(X_data)
y_train=torch.from_numpy(y_data)
training_dataset = torch.utils.data.TensorDataset(X_train, y_train)
train_loader = torch.utils.data.DataLoader(training_dataset, batch_size=50, shuffle=False)
```
However when training the model, I get the following error: RuntimeError: Given groups=1, weight of size 24 3 5, expected input[708, 256, 3] to have 3 channels, but got 256 channels instead I suppose this is due to the position of the channel? In Tensorflow, the channel position is at the end, but in PyTorch the format is "Batch Size x Channel x Height x Width"? So how do I swap the positions in the x_train tensor to match the expected format in the dataloader?
```
class TwoLayerNet(torch.nn.Module):
    def __init__(self):
        super(TwoLayerNet,self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv1d(3, 3*8, kernel_size=5, stride=1),
            nn.Sigmoid(),
            nn.AvgPool1d(kernel_size=2, stride=0))
        self.conv2 = nn.Sequential(
            nn.Conv1d(3*8, 12, kernel_size=5, stride=1),
            nn.Sigmoid(),
            nn.AvgPool1d(kernel_size=2, stride = 0))
        #self.drop_out = nn.Dropout()

        self.fc1 = nn.Linear(708, 732)
        self.fc2 = nn.Linear(732, 4)

    def forward(self, x):
        out = self.conv1(x)
        out = self.conv2(out)
        out = out.reshape(out.size(0), -1)
        out = self.drop_out(out)
        out = self.fc1(out)
        out = self.fc2(out)
        return out
```

Response:
Use permute.
```
X_train = torch.rand(708, 256, 3)
X_train = X_train.permute(2, 0, 1)
X_train.shape
# => torch.Size([3, 708, 256])
```


Question:
What does data.norm() < 1000 do in PyTorch? I am following the PyTorch tutorial here. It says that
```
x = torch.randn(3, requires_grad=True)

y = x * 2
while y.data.norm() < 1000:
    y = y * 2

print(y)

Out:
tensor([-590.4467,   97.6760,  921.0221])
```
Could someone explain what data.norm() does here? When I change .randn to .ones its output is tensor([ 1024., 1024., 1024.]).

Response:
It's simply the L2 norm (a.k.a Euclidean norm) of the tensor. Below is a reproducible illustration:
```

```python
x = torch.randn(3, requires_grad=True)
```

```python
y = x * 2
```

```python
y.data
#Output
#tensor([-1.2510, -0.6302,  1.2898])
```

```python
y.data.norm()
#Output
#tensor(1.9041)
#
## computing the norm using elementary operations
```

```python
torch.sqrt(torch.sum(torch.pow(y, 2)))
#Output
#tensor(1.9041)
#```
#Explanation: First, it takes a square of every element in the input tensor x, then it sums them together, and finally it takes a square root of the resulting sum. All in all, these operations compute the so-called L2 or Euclidean norm.
```


Question: Difference between Tensorflow's tf.keras.layers.Dense and PyTorch's torch.nn.Linear? I have a quick (and possibly silly) question about how Tensorflow defines its Linear layer. Within PyTorch, a Linear (or Dense) layer is defined as, y = x A^T + b where A and b are the weight matrix and bias vector for a Linear layer (see here). However, I can't precisely find an equivalent equation for Tensorflow! Is it the same as PyTorch or is it just y = x A + b ? Thank you in advance!

Response:
If we set activation to None in the dense layer in keras API, then they are technically equivalent. Tensorflow's
```
tf.keras.layers.Dense(..., activation=None)
```
According to the doc, more study here. activation: Activation function to use. If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x). And in PyTorch's src.
```
torch.nn.Linear
```
They are now equal at this point. A linear transformation to the incoming data: y = x*W^T + b. See the following more concrete equivalent implementation of these two. In PyTorch, we do
```
class Network(torch.nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.fc1 = torch.nn.Linear(5, 30)
    def forward(self, state):
        return self.fc1(state)
```
or,
```
trd = torch.nn.Linear(in_features = 3, out_features = 30)
y = trd(torch.ones(5, 3))
print(y.size())
# torch.Size([5, 30])
```
Its equivalent tf implementation would be
```
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(30, input_shape=(5,), activation=None))
```
or,
```
tfd = tf.keras.layers.Dense(30, input_shape=(3,), activation=None)
x = tfd(tf.ones(shape=(5, 3)))
print(x.shape)
# (5, 30)
```


Question:
PyTorch: manually setting weight parameters with numpy array for GRU / LSTM I'm trying to fill up GRU/LSTM with manually defined parameters in pytorch. I have numpy arrays for parameters with shapes as defined in their documentation (https://pytorch.org/docs/stable/nn.html#torch.nn.GRU). It seems to work but I'm not sure whether the returned values are correct. Is this a right way to fill up GRU/LSTM with numpy parameters?
```
gru = nn.GRU(input_size, hidden_size, num_layers,
              bias=True, batch_first=False, dropout=dropout, bidirectional=bidirectional)

def set_nn_wih(layer, parameter_name, w, l0=True):
    param = getattr(layer, parameter_name)
    if l0:
        for i in range(3*hidden_size):
            param.data[i] = w[i*input_size:(i+1)*input_size]
    else:
        for i in range(3*hidden_size):
            param.data[i] = w[i*num_directions*hidden_size:(i+1)*num_directions*hidden_size]

def set_nn_whh(layer, parameter_name, w):
    param = getattr(layer, parameter_name)
    for i in range(3*hidden_size):
        param.data[i] = w[i*hidden_size:(i+1)*hidden_size]

l0=True

for i in range(num_directions):
    for j in range(num_layers):
        if j == 0:
            wih = w0[i, :, :3*input_size]
            whh = w0[i, :, 3*input_size:]  # check
            l0=True
        else:
            wih = w[j-1, i, :, :num_directions*3*hidden_size]
            whh = w[j-1, i, :, num_directions*3*hidden_size:]
            l0=False

        if i == 0:
            set_nn_wih(
                gru, "weight_ih_l{}".format(j), torch.from_numpy(wih.flatten()),l0)
            set_nn_whh(
                gru, "weight_hh_l{}".format(j), torch.from_numpy(whh.flatten()))
        else:
            set_nn_wih(
                gru, "weight_ih_l{}_reverse".format(j), torch.from_numpy(wih.flatten()),l0)
            set_nn_whh(
                gru, "weight_hh_l{}_reverse".format(j), torch.from_numpy(whh.flatten()))

y, hn = gru(x_t, h_t)
```
numpy arrays are defined as following:
```
rng = np.random.RandomState(313)
w0 = rng.randn(num_directions, hidden_size, 3*(input_size +
               hidden_size)).astype(np.float32)
w = rng.randn(max(1, num_layers-1), num_directions, hidden_size,
              3*(num_directions*hidden_size + hidden_size)).astype(np.float32)
```

Response:
That is a good question, and you already give a decent answer. However, it reinvents the wheel - there is a very elegant Pytorch internal routine that will allow you to do the same without as much effort - and one that is applicable for any network. The core concept here is PyTorch's state_dict. The state dictionary effectively contains the parameters organized by the tree-structure given by the relationship of the nn.Modules and their submodules, etc. The short answer If you only want the code to load a value into a tensor using the state_dict, then try this line (where the dict contains a valid state_dict):
```
`model.load_state_dict(dict, strict=False)`
```
where strict=False is crucial if you want to load only some parameter values. The long answer - including an introduction to PyTorch's state_dict Here's an example of how a state dict looks for a GRU (I chose input_size = hidden_size = 2 so that I can print the entire state dict):
```
rnn = torch.nn.GRU(2, 2, 1)
rnn.state_dict()
# Out[10]:
#     OrderedDict([('weight_ih_l0', tensor([[-0.0023, -0.0460],
#                         [ 0.3373,  0.0070],
#                         [ 0.0745, -0.5345],
#                         [ 0.5347, -0.2373],
#                         [-0.2217, -0.2824],
#                         [-0.2983,  0.4771]])),
#                 ('weight_hh_l0', tensor([[-0.2837, -0.0571],
#                         [-0.1820,  0.6963],
#                         [ 0.4978, -0.6342],
#                         [ 0.0366,  0.2156],
#                         [ 0.5009,  0.4382],
#                         [-0.7012, -0.5157]])),
#                 ('bias_ih_l0',
#                 tensor([-0.2158, -0.6643, -0.3505, -0.0959, -0.5332, -0.6209])),
#                 ('bias_hh_l0',
#                 tensor([-0.1845,  0.4075, -0.1721, -0.4893, -0.2427,  0.3973]))])
```
So the state_dict all the parameters of the network. If we have "nested" nn.Modules, we get the tree represented by the parameter names:
```
class MLP(torch.nn.Module):
    def __init__(self):
        torch.nn.Module.__init__(self)
        self.lin_a = torch.nn.Linear(2, 2)
        self.lin_b = torch.nn.Linear(2, 2)

mlp = MLP()
mlp.state_dict()
#    Out[23]:
#        OrderedDict([('lin_a.weight', tensor([[-0.2914,  0.0791],
#                            [-0.1167,  0.6591]])),
#                    ('lin_a.bias', tensor([-0.2745, -0.1614])),
#                    ('lin_b.weight', tensor([[-0.4634, -0.2649],
#                            [ 0.4552,  0.3812]])),
#                    ('lin_b.bias', tensor([ 0.0273, -0.1283]))])

class NestedMLP(torch.nn.Module):
    def __init__(self):
        torch.nn.Module.__init__(self)
        self.mlp_a = MLP()
        self.mlp_b = MLP()

n_mlp = NestedMLP()
n_mlp.state_dict()
#   Out[26]:
#        OrderedDict([('mlp_a.lin_a.weight', tensor([[ 0.2543,  0.3412],
#                            [-0.1984, -0.3235]])),
#                    ('mlp_a.lin_a.bias', tensor([ 0.2480, -0.0631])),
#                    ('mlp_a.lin_b.weight', tensor([[-0.4575, -0.6072],
#                            [-0.0100,  0.5887]])),
#                    ('mlp_a.lin_b.bias', tensor([-0.3116,  0.5603])),
#                    ('mlp_b.lin_a.weight', tensor([[ 0.3722,  0.6940],
#                            [-0.5120,  0.5414]])),
#                    ('mlp_b.lin_a.bias', tensor([0.3604, 0.0316])),
#                    ('mlp_b.lin_b.weight', tensor([[-0.5571,  0.0830],
#                            [ 0.5230, -0.1020]])),
#                    ('mlp_b.lin_b.bias', tensor([ 0.2156, -0.2930]))])
```
So - what if you want to not extract the state dict, but change it - and thereby the network's parameters? Use nn.Module.load_state_dict(state_dict, strict=True) (link to the docs) This method allows you to load an entire state_dict with arbitrary values into an instantiated model of the same kind as long as the keys (i.e. the parameter names) are correct and the values (i.e. the parameters) are torch.tensors of the right shape. If the strict kwarg is set to True (the default), the dict you load has to exactly match the original state dict, except for the values of the parameters. That is, there has to be one new value for each parameter. For the GRU example above, we need a tensor of the correct size (and the correct device, btw) for each of 'weight_ih_l0', 'weight_hh_l0', 'bias_ih_l0', 'bias_hh_l0'. As we sometimes only want to load some values (as I think you want to do), we can set the strict kwarg to False - and we can then load only partial state dicts, as e.g. one that only contains parameter values for 'weight_ih_l0'. As a practical advice, I'd simply create the model you want to load values into, and then print the state dict (or at least a list of the keys and the respective tensor sizes)
```
print([k, v.shape for k, v in model.state_dict().items()])
```
That tells you what the exact name of the parameter is you want to change. You then simply create a state dict with the respective parameter name and tensor, and load it:
```
from dollections import OrderedDict
new_state_dict = OrderedDict({'tensor_name_retrieved_from_original_dict': new_tensor_value})
model.load_state_dict(new_state_dict, strict=False)
```


Question: Understanding input shape to PyTorch LSTM This seems to be one of the most common questions about LSTMs in PyTorch, but I am still unable to figure out what should be the input shape to PyTorch LSTM. Even after following several posts (1, 2, 3) and trying out the solutions, it doesn't seem to work. Background: I have encoded text sequences (variable length) in a batch of size 12 and the sequences are padded and packed using pad_packed_sequence functionality. MAX_LEN for each sequence is 384 and each token (or word) in the sequence has a dimension of 768. Hence my batch tensor could have one of the following shapes: [12, 384, 768] or [384, 12, 768]. The batch will be my input to the PyTorch rnn module (lstm here). According to the PyTorch documentation for LSTMs, its input dimensions are (seq_len, batch, input_size) which I understand as following. seq_len - the number of time steps in each input stream (feature vector length). batch - the size of each batch of input sequences. input_size - the dimension for each input token or time step. lstm = nn.LSTM(input_size=?, hidden_size=?, batch_first=True) What should be the exact input_size and hidden_size values here?

Response:
You have explained the structure of your input, but you haven't made the connection between your input dimensions and the LSTM's expected input dimensions. Let's break down your input (assigning names to the dimensions): batch_size: 12 seq_len: 384 input_size / num_features: 768 That means the input_size of the LSTM needs to be 768. The hidden_size is not dependent on your input, but rather how many features the LSTM should create, which is then used for the hidden state as well as the output, since that is the last hidden state. You have to decide how many features you want to use for the LSTM. Finally, for the input shape, setting batch_first=True requires the input to have the shape [batch_size, seq_len, input_size], in your case that would be [12, 384, 768].
```py
import torch
import torch.nn as nn

# Size: [batch_size, seq_len, input_size]
input = torch.randn(12, 384, 768)

lstm = nn.LSTM(input_size=768, hidden_size=512, batch_first=True)

output, _ = lstm(input)
output.size()  # => torch.Size([12, 384, 512])
```


Question:
Tokens to Words mapping in the tokenizer decode step huggingface? Is there a way to know the mapping from the tokens back to the original words in the tokenizer.decode() function? For example:
```py
from transformers.tokenization_roberta import RobertaTokenizer

tokenizer = RobertaTokenizer.from_pretrained('roberta-large', do_lower_case=True)

str = "This is a tokenization example"
tokenized = tokenizer.tokenize(str)
## ['this', 'Ġis', 'Ġa', 'Ġtoken', 'ization', 'Ġexample']

encoded = tokenizer.encode_plus(str)
## encoded['input_ids']=[0, 42, 16, 10, 19233, 1938, 1246, 2]

decoded = tokenizer.decode(encoded['input_ids'])
## '<s> this is a tokenization example</s>'
```
And the objective is to have a function that maps each token in the decode process to the correct input word, for here it will be: desired_output = [[1],[2],[3],[4,5],[6]] As this corresponds to id 42, while token and ization corresponds to ids [19244,1938] which are at indexes 4,5 of the input_ids array.

Response:
transformers version>=2.9.0: The FastTokenizers return a BatchEnconding object that you can utilize:
```py
from transformers import RobertaTokenizerFast

tokenizer = RobertaTokenizerFast.from_pretrained('roberta-large')

example = "This is a tokenization example"

enc = tokenizer(example, add_special_tokens=False)

desired_output = []

#BatchEncoding.word_ids returns a list mapping words to tokens
for w_idx in set(enc.word_ids()):
    #BatchEncoding.word_to_tokens tells us which and how many tokens are used for the specific word
    start, end = enc.word_to_tokens(w_idx)
    # we add +1 because you wanted to start with 1 and not with 0
    start+=1
    end+=1
    desired_output.append(list(range(start,end)))
```
Output:
```
[[1], [2], [3], [4, 5], [6]]
```
transformers version<2.9.0: As far as I know, there is no built-in method for that, but you can create one by yourself:
```py
from transformers.tokenization_roberta import RobertaTokenizer

tokenizer = RobertaTokenizer.from_pretrained('roberta-large', do_lower_case=True)

example = "This is a tokenization example"

print({x : tokenizer.encode(x, add_special_tokens=False, add_prefix_space=True) for x in example.split()})
```
Output:
```
{'This': [42], 'is': [16], 'a': [10], 'tokenization': [19233, 1938], 'example': [1246]}
```
To get exactly your desired output, you have to work with a list comprehension:
```py
#start index because the number of special tokens is fixed for each model (but be aware of single sentence input and pairwise sentence input)
idx = 1

enc =[tokenizer.encode(x, add_special_tokens=False, add_prefix_space=True) for x in example.split()]

desired_output = []

for token in enc:
    tokenoutput = []
    for ids in token:
      tokenoutput.append(idx)
      idx +=1
    desired_output.append(tokenoutput)

print(desired_output)
```
Output:
```
[[1], [2], [3], [4, 5], [6]]
```


Question:
converting list of tensors to tensors pytorch I have list of tensor where each tensor has a different size. How can I convert this list of tensors into a tensor using PyTorch? For instance,
```
x[0].size() == torch.Size([4, 8])
x[1].size() == torch.Size([4, 7])  # different shapes!
```
This:
```
torch.tensor(x)
```
Gives the error: ValueError: only one element tensors can be converted to Python scalars

Response:
You might be looking for cat. However, tensors cannot hold variable length data. for example, here we have a list with two tensors that have different sizes(in their last dim(dim=2)) and we want to create a larger tensor consisting of both of them, so we can use cat and create a larger tensor containing both of their data. also note that you can't use cat with half tensors on cpu as of right now so you should convert them to float, do the concatenation and then convert back to half
```
import torch

a = torch.arange(8).reshape(2, 2, 2)
b = torch.arange(12).reshape(2, 2, 3)
my_list = [a, b]
my_tensor = torch.cat([a, b], dim=2)
print(my_tensor.shape) #torch.Size([2, 2, 5])
```
you haven't explained your goal so another option is to use pad_sequence like this:
```
from torch.nn.utils.rnn import pad_sequence
a = torch.ones(25, 300)
b = torch.ones(22, 300)
c = torch.ones(15, 300)
pad_sequence([a, b, c]).size() #torch.Size([25, 3, 300])
```
edit: in this particular case, you can use torch.cat([x.float() for x in sequence], dim=1).half()


Question: How to convert a pytorch tensor of ints to a tensor of booleans? I would like to cast a tensor of ints to a tensor of booleans. Specifically I would like to be able to have a function which transforms tensor([0,10,0,16]) to tensor([0,1,0,1]) This is trivial in Tensorflow by just using tf.cast(x,tf.bool). I want the cast to change all ints greater than 0 to a 1 and all ints equal to 0 to a 0. This is the equivalent of !! in most languages. Since pytorch does not seem to have a dedicated boolean type to cast to, what is the best approach here? Edit: I am looking for a vectorized solution opposed to looping through each element.

Response:
What you're looking for is to generate a boolean mask for the given integer tensor. For this, you can simply check for the condition: "whether the values in the tensor are greater than 0" using simple comparison operator (>) or using torch.gt(), which would then give us the desired result.
```
# input tensor

```python
t
#Output
#tensor([ 0, 10,  0, 16])
#
## generate the needed boolean mask
```

```python
t > 0
#Output
#tensor([0, 1, 0, 1], dtype=torch.uint8)
#```
#```
## sanity check
```

```python
mask = t > 0
```

```python
mask.type()
#Output
#'torch.ByteTensor'
#```
#Note: In PyTorch version 1.4+, the above operation would return 'torch.BoolTensor'
#```
```

```python
t > 0
#Output
#tensor([False,  True, False,  True])
#
## alternatively, use `torch.gt()` API
```

```python
torch.gt(t, 0)
#Output
#tensor([False,  True, False,  True])
#```
#If you indeed want single bits (either 0s or 1s), cast it using:
#```
```

```python
(t > 0).type(torch.uint8)
#Output
#tensor([0, 1, 0, 1], dtype=torch.uint8)
#
## alternatively, use `torch.gt()` API
```

```python
torch.gt(t, 0).int()
#Output
#tensor([0, 1, 0, 1], dtype=torch.int32)
#```
#The reason for this change has been discussed in this feature-request issue: issues/4764 - Introduce torch.BoolTensor ... TL;DR: Simple one liner
#```
#t.bool().int()
#```
```


Question: How to check if a model is in train or eval mode in PyTorch? How to check from within a model if it is currently in train or eval mode?

Response:
From the Pytorch forum, with a small tweak: use
```
if self.training:
    # it's in train mode
else:
    # it's in eval mode
```
Always better to have a stack overflow answer than to look at forums. Explanation about the modes


Question:
TypeError: can't convert np.ndarray of type numpy.object_ How to convert a numpy array of dtype=object to torch Tensor?
```
array([
   array([0.5, 1.0, 2.0], dtype=float16),
   array([4.0, 6.0, 8.0], dtype=float16)
], dtype=object)
```

Response:
It is difficult to answer properly since you do not show us how you try to do it. From your error message I can see that you try to convert a numpy array containing objects to a torch tensor. This does not work, you will need a numeric data type:
```
import torch
import numpy as np

# Your test array without 'dtype=object'
a = np.array([
    np.array([0.5, 1.0, 2.0], dtype=np.float16),
    np.array([4.0, 6.0, 8.0], dtype=np.float16),
])

b = torch.from_numpy(a)

print(a.dtype) # This should not be 'object'
print(b)
```
Output
```
float16
tensor([[0.5000, 1.0000, 2.0000],
        [4.0000, 6.0000, 8.0000]], dtype=torch.float16)
```


Question:
Why does autograd not produce gradient for intermediate variables? trying to wrap my head around how gradients are represented and how autograd works:
```
import torch
from torch.autograd import Variable

x = Variable(torch.Tensor([2]), requires_grad=True)
y = x * x
z = y * y

z.backward()

print(x.grad)
#Variable containing:
#32
#[torch.FloatTensor of size 1]

print(y.grad)
#None
```
Why does it not produce a gradient for y? If y.grad = dz/dy, then shouldn't it at least produce a variable like y.grad = 2*y?

Response:
By default, gradients are only retained for leaf variables. non-leaf variables' gradients are not retained to be inspected later. This was done by design, to save memory. -soumith chintala See: https://discuss.pytorch.org/t/why-cant-i-see-grad-of-an-intermediate-variable/94 Option 1: Call y.retain_grad()
```
x = Variable(torch.Tensor([2]), requires_grad=True)
y = x * x
z = y * y

y.retain_grad()

z.backward()

print(y.grad)
#Variable containing:
# 8
#[torch.FloatTensor of size 1]
```
Source: https://discuss.pytorch.org/t/why-cant-i-see-grad-of-an-intermediate-variable/94/16 Option 2: Register a hook, which is basically a function called when that gradient is calculated. Then you can save it, assign it, print it, whatever...
```
from __future__ import print_function
import torch
from torch.autograd import Variable

x = Variable(torch.Tensor([2]), requires_grad=True)
y = x * x
z = y * y

y.register_hook(print) ## this can be anything you need it to be

z.backward()
```
output:
```
Variable containing:  8 [torch.FloatTensor of size 1
```
Source: https://discuss.pytorch.org/t/why-cant-i-see-grad-of-an-intermediate-variable/94/2 Also see: https://discuss.pytorch.org/t/why-cant-i-see-grad-of-an-intermediate-variable/94/7


Question: PyTorch `torch.no_grad` vs `torch.inference_mode` PyTorch has new functionality torch.inference_mode as of v1.9 which is "analogous to torch.no_grad... Code run under this mode gets better performance by disabling view tracking and version counter bumps." If I am just evaluating my model at test time (i.e. not training), is there any situation where torch.no_grad is preferable to torch.inference_mode? I plan to replace every instance of the former with the latter, and I expect to use runtime errors as a guardrail (i.e. I trust that any issue would reveal itself as a runtime error, and if it doesn't surface as a runtime error then I assume it is indeed preferable to use torch.inference_mode). More details on why inference mode was developed are mentioned in the PyTorch Developer Podcast.

Response: Yes, torch.inference_mode is indeed preferable to torch.no_grad in all situations where inference mode does not throw a runtime error. Check here.