Question: How to use `stack()` in PyTorch? How do I use torch.stack() to stack two tensors with shapes a.shape = (2, 3, 4) and b.shape = (2, 3) without an in-place operation?

Response:
Stacking requires same number of dimensions. One way would be to unsqueeze and stack. For example:
```
a.size()  # 2, 3, 4
b.size()  # 2, 3
b = torch.unsqueeze(b, dim=2)  # 2, 3, 1
# torch.unsqueeze(b, dim=-1) does the same thing

torch.stack([a, b], dim=2)  # 2, 3, 5
```


Question: What's the difference between "hidden" and "output" in PyTorch LSTM? I'm having trouble understanding the documentation for PyTorch's LSTM module (and also RNN and GRU, which are similar). Regarding the outputs, it says: Outputs: output, (h_n, c_n) output (seq_len, batch, hidden_size * num_directions): tensor containing the output features (h_t) from the last layer of the RNN, for each t. If a torch.nn.utils.rnn.PackedSequence has been given as the input, the output will also be a packed sequence. h_n (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t=seq_len c_n (num_layers * num_directions, batch, hidden_size): tensor containing the cell state for t=seq_len It seems that the variables output and h_n both give the values of the hidden state. Does h_n just redundantly provide the last time step that's already included in output, or is there something more to it than that?

Response: I made a diagram. The names follow the PyTorch docs, although I renamed num_layers to w. output comprises all the hidden states in the last layer ("last" depth-wise, not time-wise). (h_n, c_n) comprises the hidden states after the last timestep, t = n, so you could potentially feed them into another LSTM. The batch dimension is not included.


Question:
Convert integer to pytorch tensor of binary bits Given an number and an encoding length, how can I convert the number to its binary representation as a tensor? Eg, given the number 6 and width 8, how can I obtain the tensor:
```
(0, 0, 0, 0, 0, 1, 1, 0)
```

Response:
```py
def binary(x, bits):
    mask = 2**torch.arange(bits).to(x.device, x.dtype)
    return x.unsqueeze(-1).bitwise_and(mask).ne(0).byte()
```
If you wanna reverse the order of bits, use it with torch.arange(bits-1,-1,-1) instead.


Question:
How can I use Numba for Pytorch tensors? I am new to Numba and I need to use Numba to speed up some Pytorch functions. But I find even a very simple function does not work :(
```
import torch
import numba

@numba.njit()
def vec_add_odd_pos(a, b):
    res = 0.
    for pos in range(len(a)):
        if pos % 2 == 0:
            res += a[pos] + b[pos]
    return res

x = torch.tensor([3, 4, 5.])
y = torch.tensor([-2, 0, 1.])
z = vec_add_odd_pos(x, y)
```
But the following error appears
```
def vec_add_odd_pos(a, b):
    res = 0.
    ^

This error may have been caused by the following argument(s):
- argument 0: cannot determine Numba type of <class 'torch.Tensor'>
- argument 1: cannot determine Numba type of <class 'torch.Tensor'>
```
Can anyone help me? A link with more examples would be also appreciated. Thanks.

Response: Pytorch now exposes an interface on GPU tensors which can be consumed by numba directly: numba.cuda.as_cuda_array(tensor) The test script provides a few usage examples: https://github.com/pytorch/pytorch/blob/master/test/test_numba_integration.py


Question: Any pytorch tools to monitor neural network's training? Are there any tools to monitor network's training in PyTorch? Like tensorboard in tensorflow.

Response: PyTorch 1.1.0 supports TensorBoard natively with torch.utils.tensorboard. The API is very similar to tensorboardX. See the documentation for more details.


Question: Pytorch beginner : tensor.new method everyone, I have a small question. What is the purpose of the method tensor.new(..) in Pytorch, I didn't find anything in the documentation. It looks like it creates a new Tensor (like the name suggests), but why we don't just use torch.Tensor constructors instead of using this new method that requires an existing tensor. Thank you in advance.

Response: As the documentation of tensor.new() says: Constructs a new tensor of the same data type as self tensor. Also note: For CUDA tensors, this method will create new tensor on the same device as this tensor.


Question: Is there a `tensor` operation or function in Pytorch that works like cv2.dilate in OpenCV? I built several masks through a network. These masks are stored in a torch.tensor variable. I would like to do a cv2.dilate like operation on every channel of the tensor. I know there is a way that convert the tensor to numpy.ndarray and then apply cv2.dilate to every channel using a for loop. But since there are about 32 channels, this method might slow down the forward operation in the network.

Response:
I think dilate is essentially conv2d operation in torch. See the code below
```py
import cv2
import numpy as np
import torch

im = np.array([ [0, 0, 0, 0, 0],
                [0, 1, 0, 0, 0],
                [0, 1, 1, 0, 0],
                [0, 0, 0, 1, 0],
                [0, 0, 0, 0, 0] ], dtype=np.float32)
kernel = np.array([ [1, 1, 1],
                    [1, 1, 1],
                    [1, 1, 1] ], dtype=np.float32)
print(cv2.dilate(im, kernel))
# [[1. 1. 1. 0. 0.]
#  [1. 1. 1. 1. 0.]
#  [1. 1. 1. 1. 1.]
#  [1. 1. 1. 1. 1.]
#  [0. 0. 1. 1. 1.]]
im_tensor = torch.Tensor(np.expand_dims(np.expand_dims(im, 0), 0)) # size:(1, 1, 5, 5)
kernel_tensor = torch.Tensor(np.expand_dims(np.expand_dims(kernel, 0), 0)) # size: (1, 1, 3, 3)
torch_result = torch.clamp(torch.nn.functional.conv2d(im_tensor, kernel_tensor, padding=(1, 1)), 0, 1)
print(torch_result)
# tensor([[[[1., 1., 1., 0., 0.],
#           [1., 1., 1., 1., 0.],
#           [1., 1., 1., 1., 1.],
#           [1., 1., 1., 1., 1.],
#           [0., 0., 1., 1., 1.]]]])
```


Question:
How to add a new dimension to a PyTorch tensor? In NumPy, I would do
```py
a = np.zeros((4, 5, 6))
a = a[:, :, np.newaxis, :]
assert a.shape == (4, 5, 1, 6)
```
How to do the same in PyTorch?

Response:
```
a = torch.zeros(4, 5, 6)
a = a[:, :, None, :]
assert a.shape == (4, 5, 1, 6)
```


Question: How do I initialize weights in PyTorch? How do I initialize weights and biases of a network (via e.g. He or Xavier initialization)?

Response:
Single layer To initialize the weights of a single layer, use a function from torch.nn.init. For instance:
```
conv1 = torch.nn.Conv2d(...)
torch.nn.init.xavier_uniform(conv1.weight)
```
Alternatively, you can modify the parameters by writing to conv1.weight.data (which is a torch.Tensor). Example:
```
conv1.weight.data.fill_(0.01)
```
The same applies for biases:
```
conv1.bias.data.fill_(0.01)
```
nn.Sequential or custom nn.Module Pass an initialization function to torch.nn.Module.apply. It will initialize the weights in the entire nn.Module recursively. apply(fn): Applies fn recursively to every submodule (as returned by .children()) as well as self. Typical use includes initializing the parameters of a model (see also torch-nn-init). Example:
```
def init_weights(m):
    if isinstance(m, nn.Linear):
        torch.nn.init.xavier_uniform(m.weight)
        m.bias.data.fill_(0.01)

net = nn.Sequential(nn.Linear(2, 2), nn.Linear(2, 2))
net.apply(init_weights)
```


Question:
pytorch model summary - forward func has more than one argument I am using torch summary
```
from torchsummary import summary
```
I want to pass more than one argument when printing the model summary, but the examples mentioned here: Model summary in pytorch taken only one argument. for e.g.:
```
model = Network().to(device)
summary(model,(1,28,28))
```
The reason is that the forward function takes two arguments as input, e.g.:
```
def forward(self, img1, img2):
```
How do I pass two arguments here?

Response:
You can use the example given here: pytorch summary multiple inputs
```
summary(model, [(1, 16, 16), (1, 28, 28)])
```


Question:
How do you alter the size of a Pytorch Dataset? [duplicate] This question already has answers here: Taking subsets of a pytorch dataset (2 answers) Closed 3 years ago. Say I am loading MNIST from torchvision.datasets.MNIST, but I only want to load in 10000 images total, how would I slice the data to limit it to only some number of data points? I understand that the DataLoader is a generator yielding data in the size of the specified batch size, but how do you slice datasets?
```
tr = datasets.MNIST('../data', train=True, download=True, transform=transform)
te = datasets.MNIST('../data', train=False, transform=transform)
train_loader = DataLoader(tr, batch_size=args.batch_size, shuffle=True, num_workers=4, **kwargs)
test_loader = DataLoader(te, batch_size=args.batch_size, shuffle=True, num_workers=4, **kwargs)
```

Response:
You can use torch.utils.data.Subset() e.g. for the first 10,000 elements:
```py
import torch.utils.data as data_utils

indices = torch.arange(10000)
tr_10k = data_utils.Subset(tr, indices)
```


Question: How do I display a single image in PyTorch? How do I display a PyTorch Tensor of shape (3, 224, 224) representing a 224x224 RGB image? Using plt.imshow(image) gives the error: TypeError: Invalid dimensions for image data

Response:
Given a Tensor representing the image, use .permute() to put the channels as the last dimension when passing them to matplotlib:
```
plt.imshow(tensor_image.permute(1, 2, 0))
```
Note: permute does not copy or allocate memory, and from_numpy() doesn't either.


Question: How do I check if PyTorch is using the GPU? How do I check if PyTorch is using the GPU? The nvidia-smi command can detect GPU activity, but I want to check it directly from inside a Python script.

Response:
These functions should help:
```

```python
import torch
```

```python
torch.cuda.is_available()
#Output
#True
```

```python
torch.cuda.device_count()
#Output
#1
```

```python
torch.cuda.current_device()
#Output
#0
```

```python
torch.cuda.device(0)
#Output
#<torch.cuda.device at 0x7efce0b03be0>
```

```python
torch.cuda.get_device_name(0)
#Output
#'GeForce GTX 950M'
#```
#This tells us: CUDA is available and can be used by one device. Device 0 refers to the GPU GeForce GTX 950M, and it is currently chosen by PyTorch.
```


Question: How to do gradient clipping in pytorch? What is the correct way to perform gradient clipping in pytorch? I have an exploding gradients problem.

Response:
A more complete example from here:
```
optimizer.zero_grad()
loss, hidden = model(data, hidden, targets)
loss.backward()

torch.nn.utils.clip_grad_norm_(model.parameters(), args.clip)
optimizer.step()
```


Question:
PyTorch RuntimeError: DataLoader worker (pid(s) 15332) exited unexpectedly I am a beginner at PyTorch and I am just trying out some examples on this webpage. But I can't seem to get the 'super_resolution' program running due to this error: RuntimeError: DataLoader worker (pid(s) 15332) exited unexpectedly I searched the Internet and found that some people suggest setting num_workers to 0. But if I do that, the program tells me that I am running out of memory (either with CPU or GPU): RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 9663676416 bytes. Buy new RAM! or RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 4.00 GiB total capacity; 2.03 GiB already allocated; 0 bytes free; 2.03 GiB reserved in total by PyTorch) How do I fix this? I am using python 3.8 on Win10(64bit) and pytorch 1.4.0. More complete error messages (--cuda means using GPU, --threads x means passing x to the num_worker parameter): with command line arguments --upscale_factor 1 --cuda
```
File "E:\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "E:\Python38\lib\multiprocessing\queues.py", line 108, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "Z:\super_resolution\main.py", line 81, in <module>
    train(epoch)
  File "Z:\super_resolution\main.py", line 48, in train
    for iteration, batch in enumerate(training_data_loader, 1):
  File "E:\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "E:\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 841, in _next_data
    idx, data = self._get_data()
  File "E:\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 808, in _get_data
    success, data = self._try_get_data()
  File "E:\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 774, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 16596, 9376, 12756, 9844) exited unexpectedly
```
with command line arguments --upscale_factor 1 --cuda --threads 0
```
File "Z:\super_resolution\main.py", line 81, in <module>
    train(epoch)
  File "Z:\super_resolution\main.py", line 52, in train
    loss = criterion(model(input), target)
  File "E:\Python38\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "Z:\super_resolution\model.py", line 21, in forward
    x = self.relu(self.conv2(x))
  File "E:\Python38\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "E:\Python38\lib\site-packages\torch\nn\modules\conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "E:\Python38\lib\site-packages\torch\nn\modules\conv.py", line 341, in conv2d_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 4.00 GiB total capacity; 2.03 GiB already allocated; 954.35 MiB free; 2.03 GiB reserved in total by PyTorch)
```

Response: This is the solution that worked for me. it may work for other Windows users. Just remove/comment the num workers to disable parallel loads


Question:
How to batch convert sentence lengths to masks in PyTorch? For example, from
```
lens = [3, 5, 4]
```
we want to get
```
mask = [[1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1],
        [1, 1, 1, 1, 0]]
```
Both of which are torch.LongTensors.

Response:
One way that I found is:
```
torch.arange(max_len).expand(len(lens), max_len) < lens.unsqueeze(1)
```
Please share if there are better ways!


Question:
Is there an efficient way to create a random bit mask in Pytorch? I want to have a random bit mask that has some specified percent of 0s. The function I devised is:
```
def create_mask(shape, rate):
    """
    The idea is, you take a random permutations of numbers. You then mod then
    mod it by the [number of entries in the bitmask] / [percent of 0s you
    want]. The number of zeros will be exactly the rate of zeros need. You
    can clamp the values for a bitmask.
    """
    mask = torch.randperm(reduce(operator.mul, shape, 1)).float().cuda()
    # Mod it by the percent to get an even dist of 0s.
    mask = torch.fmod(mask, reduce(operator.mul, shape, 1) / rate)
    # Anything not zero should be put to 1
    mask = torch.clamp(mask, 0, 1)
    return mask.view(shape)
```
To illustrate:
```

```python
x = create_mask((10, 10), 10)
```

```python
x
```

1     1     1     1     1     1     1     1     1     1
    1     1     1     1     1     1     0     1     1     1
    0     1     1     1     1     0     1     1     1     1
    0     1     1     1     1     1     1     1     1     1
    1     1     1     1     1     1     1     1     1     0
    1     1     1     1     1     1     1     1     1     1
    1     1     1     0     1     1     1     0     1     1
    0     1     1     1     1     1     1     1     1     1
    1     1     1     0     1     1     0     1     1     1
    1     1     1     1     1     1     1     1     1     1
[torch.cuda.FloatTensor of size 10x10 (GPU 0)]
```
The main issue I have with this method is it requires the rate to divide the shape. I want a function that accepts an arbitrary decimal and gives approximately rate percent of 0s in the bitmask. Furthermore, I am trying to find a relatively efficient way of doing so. Hence, I would rather not move a numpy array from the CPU to the GPU. Is there an effiecient way of doing so that allows for a decimal rate?

Response:
For anyone running into this, this will create a bitmask with approximately 80% zero's directly on GPU. (PyTorch 0.3)
```
torch.cuda.FloatTensor(10, 10).uniform_() > 0.8
```


Question: Difference between torch.flatten() and nn.Flatten() What are the differences between torch.flatten() and torch.nn.Flatten()?

Response: Flattening is available in three forms in PyTorch As a tensor method (oop style) torch.Tensor.flatten applied directly on a tensor: x.flatten(). As a function (functional form) torch.flatten applied as: torch.flatten(x). As a module (layer nn.Module) nn.Flatten(). Generally used in a model definition. All three are identical and share the same implementation, the only difference being nn.Flatten has start_dim set to 1 by default to avoid flattening the first axis (usually the batch axis). While the other two flatten from axis=0 to axis=-1 - i.e. the entire tensor - if no arguments are given.


Question:
RuntimeError: dimension out of range (expected to be in range of [-1, 0], but got 1) Im using a Pytorch Unet model to which i am feeding in a image as input and along with that i am feeding the label as the input image mask and traning the dataset on it. The Unet model i have picked up from somewhere else, and i am using the cross-entropy loss as a loss function but i get this dimension out of range error,
```
RuntimeError
Traceback (most recent call last)
<ipython-input-358-fa0ef49a43ae> in <module>()
     16 for epoch in range(0, num_epochs):
     17     # train for one epoch
---> 18     curr_loss = train(train_loader, model, criterion, epoch, num_epochs)
     19
     20     # store best loss and save a model checkpoint

<ipython-input-356-1bd6c6c281fb> in train(train_loader, model, criterion, epoch, num_epochs)
     16         # measure loss
     17         print (outputs.size(),labels.size())
---> 18         loss = criterion(outputs, labels)
     19         losses.update(loss.data[0], images.size(0))
     20

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py in     _ _call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
--> 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

<ipython-input-355-db66abcdb074> in forward(self, logits, targets)
      9         probs_flat = probs.view(-1)
     10         targets_flat = targets.view(-1)
---> 11         return self.crossEntropy_loss(probs_flat, targets_flat)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py in     __call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
  --> 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/loss.py in f orward(self, input, target)
    599         _assert_no_grad(target)
    600         return F.cross_entropy(input, target, self.weight, self.size_average,
--> 601                                self.ignore_index, self.reduce)
    602
    603

/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py in     cross_entropy(input, target, weight, size_average, ignore_index, reduce)
   1138         >>> loss.backward()
   1139     """
-> 1140     return nll_loss(log_softmax(input, 1), target, weight, size_average, ignore_index, reduce)
   1141
   1142

/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py in     log_softmax(input, dim, _stacklevel)
    784     if dim is None:
    785         dim = _get_softmax_dim('log_softmax', input.dim(),      _stacklevel)
--> 786     return torch._C._nn.log_softmax(input, dim)
    787
    788

RuntimeError: dimension out of range (expected to be in range of [-1, 0], but got 1)
```
Part of my code looks like this
```
class crossEntropy(nn.Module):
    def __init__(self, weight = None, size_average = True):
        super(crossEntropy, self).__init__()
        self.crossEntropy_loss = nn.CrossEntropyLoss(weight, size_average)

    def forward(self, logits, targets):
        probs = F.sigmoid(logits)
        probs_flat = probs.view(-1)
        targets_flat = targets.view(-1)
        return self.crossEntropy_loss(probs_flat, targets_flat)

class UNet(nn.Module):
    def __init__(self, imsize):
        super(UNet, self).__init__()
        self.imsize = imsize

        self.activation = F.relu

        self.pool1 = nn.MaxPool2d(2)
        self.pool2 = nn.MaxPool2d(2)
        self.pool3 = nn.MaxPool2d(2)
        self.pool4 = nn.MaxPool2d(2)
        self.conv_block1_64 = UNetConvBlock(4, 64)
        self.conv_block64_128 = UNetConvBlock(64, 128)
        self.conv_block128_256 = UNetConvBlock(128, 256)
        self.conv_block256_512 = UNetConvBlock(256, 512)
        self.conv_block512_1024 = UNetConvBlock(512, 1024)

        self.up_block1024_512 = UNetUpBlock(1024, 512)
        self.up_block512_256 = UNetUpBlock(512, 256)
        self.up_block256_128 = UNetUpBlock(256, 128)
        self.up_block128_64 = UNetUpBlock(128, 64)

        self.last = nn.Conv2d(64, 2, 1)

    def forward(self, x):
        block1 = self.conv_block1_64(x)
        pool1 = self.pool1(block1)

        block2 = self.conv_block64_128(pool1)
        pool2 = self.pool2(block2)

        block3 = self.conv_block128_256(pool2)
        pool3 = self.pool3(block3)

        block4 = self.conv_block256_512(pool3)
        pool4 = self.pool4(block4)

        block5 = self.conv_block512_1024(pool4)

        up1 = self.up_block1024_512(block5, block4)

        up2 = self.up_block512_256(up1, block3)

        up3 = self.up_block256_128(up2, block2)

        up4 = self.up_block128_64(up3, block1)

        return F.log_softmax(self.last(up4))
```

Response:
According to your code:
```
probs_flat = probs.view(-1)
targets_flat = targets.view(-1)
return self.crossEntropy_loss(probs_flat, targets_flat)
```
You are giving two 1d tensor to nn.CrossEntropyLoss but according to documentation, it expects:
```
Input: (N,C) where C = number of classes
Target: (N) where each value is 0 <= targets[i] <= C-1
Output: scalar. If reduce is False, then (N) instead.
```
I believe that is the cause of the problem you are encountering.


Question: Why Pytorch officially use mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225] to normalize images? In this page (https://pytorch.org/vision/stable/models.html), it says that "All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]". Shouldn't the usual mean and std of normalization be [0.5, 0.5, 0.5] and [0.5, 0.5, 0.5]? Why is it setting such strange values?

Response: Using the mean and std of Imagenet is a common practice. They are calculated based on millions of images. If you want to train from scratch on your own dataset, you can calculate the new mean and std. Otherwise, using the Imagenet pretrianed model with its own mean and std is recommended.


Question: Is there a pytorch method to check the number of cpus? I can use this torch.cuda.device_count() to check the number of GPUs. I was wondering if there was something equivalent to check the number of CPUs.

Response:
just use this :
```
os.cpu_count()
```


Question:
torch.Tensor() new() received an invalid combination of arguments - got (list, dtype=torch.dtype) I am facing a very weird error when I am trying to run the following simple line :
```
a = torch.Tensor([0,0,0],dtype = torch.int64)

TypeError: new() received an invalid combination of arguments - got (list, dtype=torch.dtype), but expected one of:
 * (*, torch.device device)
      didn't match because some of the keywords were incorrect: dtype
 * (torch.Storage storage)
 * (Tensor other)
 * (tuple of ints size, *, torch.device device)
 * (object data, *, torch.device device)
```
whereas if we look at official documentation
```
torch.tensor(data, dtype=None, device=None, requires_grad=False, pin_memory=False) → Tensor
```
Parameters data (array_like) – Initial data for the tensor. Can be a list, tuple, NumPy ndarray, scalar, and other types. dtype (torch.dtype, optional) – the desired data type of returned tensor. Default: if None, infers data type from data. Why is the code snippet not working even though the official documentation supports it?

Response: Capitalization matters -- in your top example, you have Tensor with uppercase T, but the documentation excerpt is talking about tensor with lowercase t.


Question: How can I do scatter and gather operations in NumPy? I want to implement the scatter and gather operations of Tensorflow or PyTorch in Numpy. torch.scatter torch.gather

Response: There are two built-in numpy functions that suit your request: Use np.take_along_axis to implement torch.gather Use np.put_along_axis to implement torch.scatter


Question:
How do I get the value of a tensor in PyTorch? Printing a tensor x gives:
```

```python
x = torch.tensor([3])
```

```python
print(x)
#Output
#tensor([3])
#```
#Indexing x.data gives:
#```
```

```python
x.data[0]
#Output
#tensor(3)
#```
#How do I get just a regular non-tensor value 3?
```

Response: You can use x.item() to get a Python number from a Tensor that has one element.


Question: How to split data into 3 sets (train, validation and test)? I have a pandas dataframe and I wish to divide it to 3 separate sets. I know that using train_test_split from sklearn.cross_validation, one can divide the data in two sets (train and test). However, I couldn't find any solution about splitting the data into three sets. Preferably, I'd like to have the indices of the original data. I know that a workaround would be to use train_test_split two times and somehow adjust the indices. But is there a more standard / built-in way to split the data into 3 sets instead of 2?

Response:
Numpy solution. We will shuffle the whole dataset first (df.sample(frac=1, random_state=42)) and then split our data set into the following parts: 60% - train set, 20% - validation set, 20% - test set
```

```python
train, validate, test = \
              np.split(df.sample(frac=1, random_state=42),
                       [int(.6*len(df)), int(.8*len(df))])
```

```python
train
#Output
#A         B         C         D         E
#0  0.046919  0.792216  0.206294  0.440346  0.038960
#2  0.301010  0.625697  0.604724  0.936968  0.870064
#1  0.642237  0.690403  0.813658  0.525379  0.396053
#9  0.488484  0.389640  0.599637  0.122919  0.106505
#8  0.842717  0.793315  0.554084  0.100361  0.367465
#7  0.185214  0.603661  0.217677  0.281780  0.938540
```

```python
validate
#Output
#A         B         C         D         E
#5  0.806176  0.008896  0.362878  0.058903  0.026328
#6  0.145777  0.485765  0.589272  0.806329  0.703479
```

```python
test
#Output
#A         B         C         D         E
#4  0.521640  0.332210  0.370177  0.859169  0.401087
#3  0.333348  0.964011  0.083498  0.670386  0.169619
#```
#[int(.6*len(df)), int(.8*len(df))] - is an indices_or_sections array for numpy.split(). Here is a small demo for np.split() usage - let's split 20-elements array into the following parts: 80%, 10%, 10%:
#```
```

```python
a = np.arange(1, 21)
```

```python
a
#Output
#array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20])
```

```python
np.split(a, [int(.8 * len(a)), int(.9 * len(a))])
#Output
#[array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16]),
# array([17, 18]),
# array([19, 20])]
#```
```


Question:
pandas dataframe columns scaling with sklearn I have a pandas dataframe with mixed type columns, and I'd like to apply sklearn's min_max_scaler to some of the columns. Ideally, I'd like to do these transformations in place, but haven't figured out a way to do that yet. I've written the following code that works:
```
import pandas as pd
import numpy as np
from sklearn import preprocessing

scaler = preprocessing.MinMaxScaler()

dfTest = pd.DataFrame({'A':[14.00,90.20,90.95,96.27,91.21],'B':[103.02,107.26,110.35,114.23,114.68], 'C':['big','small','big','small','small']})
min_max_scaler = preprocessing.MinMaxScaler()

def scaleColumns(df, cols_to_scale):
    for col in cols_to_scale:
        df[col] = pd.DataFrame(min_max_scaler.fit_transform(pd.DataFrame(dfTest[col])),columns=[col])
    return df

dfTest

    A   B   C
0    14.00   103.02  big
1    90.20   107.26  small
2    90.95   110.35  big
3    96.27   114.23  small
4    91.21   114.68  small

scaled_df = scaleColumns(dfTest,['A','B'])
scaled_df

A   B   C
0    0.000000    0.000000    big
1    0.926219    0.363636    small
2    0.935335    0.628645    big
3    1.000000    0.961407    small
4    0.938495    1.000000    small
```
I'm curious if this is the preferred/most efficient way to do this transformation. Is there a way I could use df.apply that would be better? I'm also surprised I can't get the following code to work:
```
bad_output = min_max_scaler.fit_transform(dfTest['A'])
```
If I pass an entire dataframe to the scaler it works:
```
dfTest2 = dfTest.drop('C', axis = 1)
good_output = min_max_scaler.fit_transform(dfTest2)
good_output
```
I'm confused why passing a series to the scaler fails. In my full working code above I had hoped to just pass a series to the scaler then set the dataframe column = to the scaled series.

Response:
I am not sure if previous versions of pandas prevented this but now the following snippet works perfectly for me and produces exactly what you want without having to use apply
```

```python
import pandas as pd
```

```python
from sklearn.preprocessing import MinMaxScaler
```

```python
scaler = MinMaxScaler()
```

```python
dfTest = pd.DataFrame({'A':[14.00,90.20,90.95,96.27,91.21],
#Output
#                           'B':[103.02,107.26,110.35,114.23,114.68],
#                           'C':['big','small','big','small','small']})
```

```python
dfTest[['A', 'B']] = scaler.fit_transform(dfTest[['A', 'B']])
```

```python
dfTest
#Output
#          A         B      C
#0  0.000000  0.000000    big
#1  0.926219  0.363636  small
#2  0.935335  0.628645    big
#3  1.000000  0.961407  small
#4  0.938495  1.000000  small
#```
```


Question: Can anyone explain me StandardScaler? I am unable to understand the page of the StandardScaler in the documentation of sklearn. Can anyone explain this to me in simple terms?

Response:
Intro I assume that you have a matrix X where each row/line is a sample/observation and each column is a variable/feature (this is the expected input for any sklearn ML function by the way -- X.shape should be [number_of_samples, number_of_features]). Core of method The main idea is to normalize/standardize i.e. μ = 0 and σ = 1 your features/variables/columns of X, individually, before applying any machine learning model. StandardScaler() will normalize the features i.e. each column of X, INDIVIDUALLY, so that each column/feature/variable will have μ = 0 and σ = 1. P.S: I find the most upvoted answer on this page, wrong. I am quoting "each value in the dataset will have the sample mean value subtracted" -- This is neither true nor correct. See also: How and why to Standardize your data: A python tutorial Example with code
```
from sklearn.preprocessing import StandardScaler
import numpy as np

# 4 samples/observations and 2 variables/features
data = np.array([[0, 0], [1, 0], [0, 1], [1, 1]])
scaler = StandardScaler()
scaled_data = scaler.fit_transform(data)

print(data)
[[0, 0],
 [1, 0],
 [0, 1],
 [1, 1]])

print(scaled_data)
[[-1. -1.]
 [ 1. -1.]
 [-1.  1.]
 [ 1.  1.]]
```
Verify that the mean of each feature (column) is 0:
```
scaled_data.mean(axis = 0)
array([0., 0.])
```
Verify that the std of each feature (column) is 1:
```
scaled_data.std(axis = 0)
array([1., 1.])
```
Appendix: The maths UPDATE 08/2020: Concerning the input parameters with_mean and with_std to False/True, I have provided an answer here: StandardScaler difference between “with_std=False or True” and “with_mean=False or True”


Question: Logistic regression python solvers' definitions I am using the logistic regression function from sklearn, and was wondering what each of the solver is actually doing behind the scenes to solve the optimization problem. Can someone briefly describe what "newton-cg", "sag", "lbfgs" and "liblinear" are doing?

Response: Well, I hope I'm not too late for the party! Let me first try to establish some intuition before digging into loads of information (warning: this is not a brief comparison, TL;DR) Introduction A hypothesis h(x), takes an input and gives us the estimated output value. This hypothesis can be as simple as a one-variable linear equation, .. up to a very complicated and long multivariate equation with respect to the type of algorithm we’re using (e.g. linear regression, logistic regression..etc). Our task is to find the best Parameters (a.k.a Thetas or Weights) that give us the least error in predicting the output. We call the function that calculates this error a Cost or Loss Function, and apparently, our goal is to minimize the error in order to get the best-predicted output! One more thing to recall is, the relation between the parameter value and its effect on the cost function (i.e. the error) looks like a bell curve (i.e. Quadratic; recall this because it’s important). So if we start at any point in that curve and keep taking the derivative (i.e. tangent line) of each point we stop at (assuming it's a univariate problem, otherwise, if we have multiple features, we take the partial derivative), we will end up at what so-called the Global Optima as shown in this image: If we take the partial derivative at the minimum cost point (i.e. global optima) we find the slope of the tangent line = 0 (then we know that we reached our target). That’s valid only if we have a Convex Cost Function, but if we don’t, we may end up stuck at what is called Local Optima; consider this non-convex function: Now you should have the intuition about the heck relationship between what we are doing and the terms: Derivative, Tangent Line, Cost Function, Hypothesis ..etc. Side Note: The above-mentioned intuition is also related to the Gradient Descent Algorithm (see later). Background Linear Approximation: Given a function, f(x), we can find its tangent at x=a. The equation of the tangent line L(x) is: L(x)=f(a)+f′(a)(x−a). Take a look at the following graph of a function and its tangent line: From this graph we can see that near x=a, the tangent line and the function have nearly the same graph. On occasion, we will use the tangent line, L(x), as an approximation to the function, f(x), near x=a. In these cases, we call the tangent line the "Linear Approximation" to the function at x=a. Quadratic Approximation: Same as a linear approximation, yet this time we are dealing with a curve where we cannot find the point near to 0 by using only the tangent line. Instead, we use the parabola as it's shown in the following graph: In order to fit a good parabola, both parabola and quadratic function should have the same value, the same first derivative, AND the same second derivative. The formula will be (just out of curiosity): Qa(x) = f(a) + f'(a)(x-a) + f''(a)(x-a)2/2 Now we should be ready to do the comparison in detail. Comparison between the methods 1. Newton’s Method Recall the motivation for the gradient descent step at x: we minimize the quadratic function (i.e. Cost Function). Newton’s method uses in a sense a better quadratic function minimisation. It's better because it uses the quadratic approximation (i.e. first AND second partial derivatives). You can imagine it as a twisted Gradient Descent with the Hessian (the Hessian is a square matrix of second-order partial derivatives of order n X n). Moreover, the geometric interpretation of Newton's method is that at each iteration one approximates f(x) by a quadratic function around xn, and then takes a step towards the maximum/minimum of that quadratic function (in higher dimensions, this may also be a saddle point). Note that if f(x) happens to be a quadratic function, then the exact extremum is found in one step. Drawbacks: It’s computationally expensive because of the Hessian Matrix (i.e. second partial derivatives calculations). It attracts to Saddle Points which are common in multivariable optimization (i.e. a point that its partial derivatives disagree over whether this input should be a maximum or a minimum point!). 2. Limited-memory Broyden–Fletcher–Goldfarb–Shanno Algorithm: In a nutshell, it is an analogue of Newton’s Method, yet here the Hessian matrix is approximated using updates specified by gradient evaluations (or approximate gradient evaluations). In other words, using estimation to the inverse Hessian matrix. The term Limited-memory simply means it stores only a few vectors that represent the approximation implicitly. If I dare say that when the dataset is small, L-BFGS relatively performs the best compared to other methods especially because it saves a lot of memory, however, there are some “serious” drawbacks such that if it is unsafeguarded, it may not converge to anything. Side note: This solver has become the default solver in sklearn LogisticRegression since version 0.22, replacing LIBLINEAR. 3. A Library for Large Linear Classification: It’s a linear classification that supports logistic regression and linear support vector machines. The solver uses a Coordinate Descent (CD) algorithm that solves optimization problems by successively performing approximate minimization along coordinate directions or coordinate hyperplanes. LIBLINEAR is the winner of the ICML 2008 large-scale learning challenge. It applies automatic parameter selection (a.k.a L1 Regularization) and it’s recommended when you have high dimension dataset (recommended for solving large-scale classification problems) Drawbacks: It may get stuck at a non-stationary point (i.e. non-optima) if the level curves of a function are not smooth. Also cannot run in parallel. It cannot learn a true multinomial (multiclass) model; instead, the optimization problem is decomposed in a “one-vs-rest” fashion, so separate binary classifiers are trained for all classes. Side note: According to Scikit Documentation: The “liblinear” solver was the one used by default for historical reasons before version 0.22. Since then, the default use is Limited-memory Broyden–Fletcher–Goldfarb–Shanno Algorithm. 4. Stochastic Average Gradient: The SAG method optimizes the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values, the SAG method achieves a faster convergence rate than black-box SG methods. It is faster than other solvers for large datasets when both the number of samples and the number of features are large. Drawbacks: It only supports L2 penalization. This is not really a drawback, but more like a comparison: although SAG is suitable for large datasets, with a memory cost of O(N), it can be less practical for very large N (as the most recent gradient evaluation for each function needs to be maintained in the memory). This is usually not a problem, but a better option would be SVRG 1, 2 which is unfortunately not implemented in scikit-learn! 5. SAGA: The SAGA solver is a variant of SAG that also supports the non-smooth penalty L1 option (i.e. L1 Regularization). This is therefore the solver of choice for sparse multinomial logistic regression. It also has a better theoretical convergence compared to SAG. Drawbacks: This is not really a drawback, but more like a comparison: SAGA is similar to SAG with regard to memory cost. That's it's suitable for large datasets, yet in edge cases where the dataset is very large, the SVRG 1, 2 would be a better option (unfortunately not implemented in scikit-learn)! Side note: According to Scikit Documentation: The SAGA solver is often the best choice. Please note the attributes "Large" and "Small" used in Scikit-Learn and in this comparison are relative. AFAIK, there is no universal unanimous and accurate definition of the dataset boundaries to be considered as "Large", "Too Large", "Small", "Too Small"...etc! Summary The following table is taken from Scikit Documentation Updated Table from the same link above (accessed 02/11/2021):


Question:
What are the pros and cons between get_dummies (Pandas) and OneHotEncoder (Scikit-learn)? I'm learning different methods to convert categorical variables to numeric for machine-learning classifiers. I came across the pd.get_dummies method and sklearn.preprocessing.OneHotEncoder() and I wanted to see how they differed in terms of performance and usage. I found a tutorial on how to use OneHotEncoder() on https://xgdgsc.wordpress.com/2015/03/20/note-on-using-onehotencoder-in-scikit-learn-to-work-on-categorical-features/ since the sklearn documentation wasn't too helpful on this feature. I have a feeling I'm not doing it correctly...but Can some explain the pros and cons of using pd.dummies over sklearn.preprocessing.OneHotEncoder() and vice versa? I know that OneHotEncoder() gives you a sparse matrix but other than that I'm not sure how it is used and what the benefits are over the pandas method. Am I using it inefficiently?
```
import pandas as pd
import numpy as np
from sklearn.datasets import load_iris
sns.set()

%matplotlib inline

#Iris Plot
iris = load_iris()
n_samples, m_features = iris.data.shape

#Load Data
X, y = iris.data, iris.target
D_target_dummy = dict(zip(np.arange(iris.target_names.shape[0]), iris.target_names))

DF_data = pd.DataFrame(X,columns=iris.feature_names)
DF_data["target"] = pd.Series(y).map(D_target_dummy)
#sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)  \
#0                  5.1               3.5                1.4               0.2
#1                  4.9               3.0                1.4               0.2
#2                  4.7               3.2                1.3               0.2
#3                  4.6               3.1                1.5               0.2
#4                  5.0               3.6                1.4               0.2
#5                  5.4               3.9                1.7               0.4

DF_dummies = pd.get_dummies(DF_data["target"])
#setosa  versicolor  virginica
#0         1           0          0
#1         1           0          0
#2         1           0          0
#3         1           0          0
#4         1           0          0
#5         1           0          0

from sklearn.preprocessing import OneHotEncoder, LabelEncoder
def f1(DF_data):
    Enc_ohe, Enc_label = OneHotEncoder(), LabelEncoder()
    DF_data["Dummies"] = Enc_label.fit_transform(DF_data["target"])
    DF_dummies2 = pd.DataFrame(Enc_ohe.fit_transform(DF_data[["Dummies"]]).todense(), columns = Enc_label.classes_)
    return(DF_dummies2)

%timeit pd.get_dummies(DF_data["target"])
#1000 loops, best of 3: 777 µs per loop

%timeit f1(DF_data)
#100 loops, best of 3: 2.91 ms per loop
```

Response:
For machine learning, you almost definitely want to use sklearn.OneHotEncoder. For other tasks like simple analyses, you might be able to use pd.get_dummies, which is a bit more convenient. Note that sklearn.OneHotEncoder has been updated in the latest version so that it does accept strings for categorical variables, as well as integers. The crux of it is that the sklearn encoder creates a function which persists and can then be applied to new data sets which use the same categorical variables, with consistent results.
```
from sklearn.preprocessing import OneHotEncoder

# Create the encoder.
encoder = OneHotEncoder(handle_unknown="ignore")
encoder.fit(X_train)    # Assume for simplicity all features are categorical.

# Apply the encoder.
X_train = encoder.transform(X_train)
X_test = encoder.transform(X_test)
```
Note how we apply the same encoder we created via X_train to the new data set X_test. Consider what happens if X_test contains different levels than X_train for one of its variables. For example, let's say X_train["color"] contains only "red" and "green", but in addition to those, X_test["color"] sometimes contains "blue". If we use pd.get_dummies, X_test will end up with an additional "color_blue" column which X_train doesn't have, and the inconsistency will probably break our code later on, especially if we are feeding X_test to an sklearn model which we trained on X_train. And if we want to process the data like this in production, where we're receiving a single example at a time, pd.get_dummies won't be of use. With sklearn.OneHotEncoder on the other hand, once we've created the encoder, we can reuse it to produce the same output every time, with columns only for "red" and "green". And we can explicitly control what happens when it encounters the new level "blue": if we think that's impossible, then we can tell it to throw an error with handle_unknown="error"; otherwise we can tell it to continue and simply set the red and green columns to 0, with handle_unknown="ignore".


Question:
Stratified Train/Test-split in scikit-learn I need to split my data into a training set (75%) and test set (25%). I currently do that with the code below:
```
X, Xt, userInfo, userInfo_train = sklearn.cross_validation.train_test_split(X, userInfo)
```
However, I'd like to stratify my training dataset. How do I do that? I've been looking into the StratifiedKFold method, but doesn't let me specifiy the 75%/25% split and only stratify the training dataset.

Response:
[update for 0.17] See the docs of sklearn.model_selection.train_test_split:
```
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    stratify=y,
                                                    test_size=0.25)
```
[/update for 0.17] There is a pull request here. But you can simply do train, test = next(iter(StratifiedKFold(...))) and use the train and test indices if you want.


Question:
LogisticRegression: Unknown label type: 'continuous' using sklearn in python I have the following code to test some of most popular ML algorithms of sklearn python library:
```
import numpy as np
from sklearn                        import metrics, svm
from sklearn.linear_model           import LinearRegression
from sklearn.linear_model           import LogisticRegression
from sklearn.tree                   import DecisionTreeClassifier
from sklearn.neighbors              import KNeighborsClassifier
from sklearn.discriminant_analysis  import LinearDiscriminantAnalysis
from sklearn.naive_bayes            import GaussianNB
from sklearn.svm                    import SVC

trainingData    = np.array([ [2.3, 4.3, 2.5],  [1.3, 5.2, 5.2],  [3.3, 2.9, 0.8],  [3.1, 4.3, 4.0]  ])
trainingScores  = np.array( [3.4, 7.5, 4.5, 1.6] )
predictionData  = np.array([ [2.5, 2.4, 2.7],  [2.7, 3.2, 1.2] ])

clf = LinearRegression()
clf.fit(trainingData, trainingScores)
print("LinearRegression")
print(clf.predict(predictionData))

clf = svm.SVR()
clf.fit(trainingData, trainingScores)
print("SVR")
print(clf.predict(predictionData))

clf = LogisticRegression()
clf.fit(trainingData, trainingScores)
print("LogisticRegression")
print(clf.predict(predictionData))

clf = DecisionTreeClassifier()
clf.fit(trainingData, trainingScores)
print("DecisionTreeClassifier")
print(clf.predict(predictionData))

clf = KNeighborsClassifier()
clf.fit(trainingData, trainingScores)
print("KNeighborsClassifier")
print(clf.predict(predictionData))

clf = LinearDiscriminantAnalysis()
clf.fit(trainingData, trainingScores)
print("LinearDiscriminantAnalysis")
print(clf.predict(predictionData))

clf = GaussianNB()
clf.fit(trainingData, trainingScores)
print("GaussianNB")
print(clf.predict(predictionData))

clf = SVC()
clf.fit(trainingData, trainingScores)
print("SVC")
print(clf.predict(predictionData))
```
The first two works ok, but I got the following error in LogisticRegression call:
```
root@ubupc1:/home/ouhma# python stack.py
LinearRegression
[ 15.72023529   6.46666667]
SVR
[ 3.95570063  4.23426243]
Traceback (most recent call last):
  File "stack.py", line 28, in <module>
    clf.fit(trainingData, trainingScores)
  File "/usr/local/lib/python2.7/dist-packages/sklearn/linear_model/logistic.py", line 1174, in fit
    check_classification_targets(y)
  File "/usr/local/lib/python2.7/dist-packages/sklearn/utils/multiclass.py", line 172, in check_classification_targets
    raise ValueError("Unknown label type: %r" % y_type)
ValueError: Unknown label type: 'continuous'
```
The input data is the same as in the previous calls, so what is going on here? And by the way, why there is a huge diference in the first prediction of LinearRegression() and SVR() algorithms (15.72 vs 3.95)?

Response:
You are passing floats to a classifier which expects categorical values as the target vector. If you convert it to int it will be accepted as input (although it will be questionable if that's the right way to do it). It would be better to convert your training scores by using scikit's labelEncoder function. The same is true for your DecisionTree and KNeighbors qualifier.
```
from sklearn import preprocessing
from sklearn import utils

lab_enc = preprocessing.LabelEncoder()
encoded = lab_enc.fit_transform(trainingScores)

```python
array([1, 3, 2, 0], dtype=int64)
```

print(utils.multiclass.type_of_target(trainingScores))

```python
continuous
```

print(utils.multiclass.type_of_target(trainingScores.astype('int')))

```python
multiclass
```

print(utils.multiclass.type_of_target(encoded))

```python
multiclass
#Output
#```
```


Question: A progress bar for scikit-learn? Is there any way to have a progress bar to the fit method in scikit-learn ? Is it possible to include a custom one with something like Pyprind ?

Response:
If you initialize the model with verbose=1 before calling fit you should get some kind of output indicating the progress. For example sklearn.ensemble.GradientBoostingClassifer(verbose=1) provides progress output that looks like this:
```
Iter       Train Loss   Remaining Time
     1           1.2811            0.71s
     2           1.2595            0.58s
     3           1.2402            0.50s
     4           1.2263            0.46s
     5           1.2121            0.43s
     6           1.1999            0.41s
     7           1.1876            0.39s
     8           1.1761            0.38s
     9           1.1673            0.37s
    10           1.1591            0.36s
    20           1.1021            0.29s
    30           1.0511            0.27s
    40           1.0116            0.25s
    50           0.9830            0.22s
    60           0.9581            0.19s
    70           0.9377            0.16s
    80           0.9169            0.14s
    90           0.9049            0.12s
   100           0.8973            0.10s
```


Question:
sklearn Logistic Regression "ValueError: Found array with dim 3. Estimator expected <= 2." I attempt to solve this problem 6 in this notebook. The question is to train a simple model on this data using 50, 100, 1000 and 5000 training samples by using the LogisticRegression model from sklearn.linear_model.
```
lr = LogisticRegression()
lr.fit(train_dataset,train_labels)
```
This is the code i trying to do and it give me the error.
```
ValueError: Found array with dim 3. Estimator expected <= 2.
```
Any idea?

Response:
scikit-learn expects 2d num arrays for the training dataset for a fit function. The dataset you are passing in is a 3d array you need to reshape the array into a 2d.
```
nsamples, nx, ny = train_dataset.shape
d2_train_dataset = train_dataset.reshape((nsamples,nx*ny))
```


Question:
Different result with roc_auc_score() and auc() I have trouble understanding the difference (if there is one) between roc_auc_score() and auc() in scikit-learn. Im tying to predict a binary output with imbalanced classes (around 1.5% for Y=1). Classifier
```
model_logit = LogisticRegression(class_weight='auto')
model_logit.fit(X_train_ridge, Y_train)
```
Roc curve
```
false_positive_rate, true_positive_rate, thresholds = roc_curve(Y_test, clf.predict_proba(xtest)[:,1])
```
AUC's
```
auc(false_positive_rate, true_positive_rate)
Out[490]: 0.82338034042531527
```
and
```
roc_auc_score(Y_test, clf.predict(xtest))
Out[493]: 0.75944737191205602
```
Somebody can explain this difference ? I thought both were just calculating the area under the ROC curve. Might be because of the imbalanced dataset but I could not figure out why. Thanks!

Response:
AUC is not always area under the curve of a ROC curve. Area Under the Curve is an (abstract) area under some curve, so it is a more general thing than AUROC. With imbalanced classes, it may be better to find AUC for a precision-recall curve. See sklearn source for roc_auc_score:
```
def roc_auc_score(y_true, y_score, average="macro", sample_weight=None):
    # <...> docstring <...>
    def _binary_roc_auc_score(y_true, y_score, sample_weight=None):
            # <...> bla-bla <...>

            fpr, tpr, tresholds = roc_curve(y_true, y_score,
                                            sample_weight=sample_weight)
            return auc(fpr, tpr, reorder=True)

    return _average_binary_score(
        _binary_roc_auc_score, y_true, y_score, average,
        sample_weight=sample_weight)
```
As you can see, this first gets a roc curve, and then calls auc() to get the area. I guess your problem is the predict_proba() call. For a normal predict() the outputs are always the same:
```
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_curve, auc, roc_auc_score

est = LogisticRegression(class_weight='auto')
X = np.random.rand(10, 2)
y = np.random.randint(2, size=10)
est.fit(X, y)

false_positive_rate, true_positive_rate, thresholds = roc_curve(y, est.predict(X))
print auc(false_positive_rate, true_positive_rate)
# 0.857142857143
print roc_auc_score(y, est.predict(X))
# 0.857142857143
```
If you change the above for this, you'll sometimes get different outputs:
```
false_positive_rate, true_positive_rate, thresholds = roc_curve(y, est.predict_proba(X)[:,1])
# may differ
print auc(false_positive_rate, true_positive_rate)
print roc_auc_score(y, est.predict(X))
```


Question:
The easiest way for getting feature names after running SelectKBest in Scikit Learn I'm trying to conduct a supervised machine-learning experiment using the SelectKBest feature of scikit-learn, but I'm not sure how to create a new dataframe after finding the best features: Let's assume I would like to conduct the experiment selecting 5 best features:
```
from sklearn.feature_selection import SelectKBest, f_classif

select_k_best_classifier = SelectKBest(score_func=f_classif, k=5).fit_transform(features_dataframe, targeted_class)
```
Now, if I add the line:
```
import pandas as pd

dataframe = pd.DataFrame(select_k_best_classifier)
```
I receive a new dataframe without feature names (only index starting from 0 to 4), but I want to create a dataframe with the new selected features, in a way like this:
```
dataframe = pd.DataFrame(fit_transofrmed_features, columns=features_names)
```
My question is how to create the features_names list? I know that I should use:
```
select_k_best_classifier.get_support()
```
Which returns an array of boolean values, where true values indices represent the column that should be selected in the original dataframe. How should I use this boolean array with the array of all features names I can get via the method feature_names = list(features_dataframe.columns.values) ?

Response:
This doesn't require loops.
```
# Create and fit selector
selector = SelectKBest(f_classif, k=5)
selector.fit(features_df, target)
# Get columns to keep and create new dataframe with those only
cols_idxs = selector.get_support(indices=True)
features_df_new = features_df.iloc[:,cols_idxs]
```


Question:
difference between StratifiedKFold and StratifiedShuffleSplit in sklearn As from the title I am wondering what is the difference between StratifiedKFold with the parameter shuffle=True
```
StratifiedKFold(n_splits=10, shuffle=True, random_state=0)
```
and StratifiedShuffleSplit
```
StratifiedShuffleSplit(n_splits=10, test_size=’default’, train_size=None, random_state=0)
```
and what is the advantage of using StratifiedShuffleSplit

Response:
In stratKFolds, each test set should not overlap, even when shuffle is included. With stratKFolds and shuffle=True, the data is shuffled once at the start, and then divided into the number of desired splits. The test data is always one of the splits, the train data is the rest. In ShuffleSplit, the data is shuffled every time, and then split. This means the test sets may overlap between the splits. See this block for an example of the difference. Note the overlap of the elements in the test sets for ShuffleSplit.
```
splits = 5

tx = range(10)
ty = [0] * 5 + [1] * 5

from sklearn.model_selection import StratifiedShuffleSplit, StratifiedKFold
from sklearn import datasets

stratKfold = StratifiedKFold(n_splits=splits, shuffle=True, random_state=42)
shufflesplit = StratifiedShuffleSplit(n_splits=splits, random_state=42, test_size=2)

print("stratKFold")
for train_index, test_index in stratKfold.split(tx, ty):
    print("TRAIN:", train_index, "TEST:", test_index)

print("Shuffle Split")
for train_index, test_index in shufflesplit.split(tx, ty):
    print("TRAIN:", train_index, "TEST:", test_index)
```
Output:
```
stratKFold
TRAIN: [0 2 3 4 5 6 7 9] TEST: [1 8]
TRAIN: [0 1 2 3 5 7 8 9] TEST: [4 6]
TRAIN: [0 1 3 4 5 6 8 9] TEST: [2 7]
TRAIN: [1 2 3 4 6 7 8 9] TEST: [0 5]
TRAIN: [0 1 2 4 5 6 7 8] TEST: [3 9]
Shuffle Split
TRAIN: [8 4 1 0 6 5 7 2] TEST: [3 9]
TRAIN: [7 0 3 9 4 5 1 6] TEST: [8 2]
TRAIN: [1 2 5 6 4 8 9 0] TEST: [3 7]
TRAIN: [4 6 7 8 3 5 1 2] TEST: [9 0]
TRAIN: [7 2 6 5 4 3 0 9] TEST: [1 8]
```
As for when to use them, I tend to use stratKFolds for any cross validation, and I use ShuffleSplit with a split of 2 for my train/test set splits. But I'm sure there are other use cases for both.


Question: Mixing categorial and continuous data in Naive Bayes classifier using scikit-learn I'm using scikit-learn in Python to develop a classification algorithm to predict the gender of certain customers. Amongst others, I want to use the Naive Bayes classifier but my problem is that I have a mix of categorical data (ex: "Registered online", "Accepts email notifications" etc) and continuous data (ex: "Age", "Length of membership" etc). I haven't used scikit much before but I suppose that that Gaussian Naive Bayes is suitable for continuous data and that Bernoulli Naive Bayes can be used for categorical data. However, since I want to have both categorical and continuous data in my model, I don't really know how to handle this. Any ideas would be much appreciated!

Response: You have at least two options: Transform all your data into a categorical representation by computing percentiles for each continuous variables and then binning the continuous variables using the percentiles as bin boundaries. For instance for the height of a person create the following bins: "very small", "small", "regular", "big", "very big" ensuring that each bin contains approximately 20% of the population of your training set. We don't have any utility to perform this automatically in scikit-learn but it should not be too complicated to do it yourself. Then fit a unique multinomial NB on those categorical representation of your data. Independently fit a gaussian NB model on the continuous part of the data and a multinomial NB model on the categorical part. Then transform all the dataset by taking the class assignment probabilities (with predict_proba method) as new features: np.hstack((multinomial_probas, gaussian_probas)) and then refit a new model (e.g. a new gaussian NB) on the new features.


Question: Should I use `random.seed` or `numpy.random.seed` to control random number generation in `scikit-learn`? I'm using scikit-learn and numpy and I want to set the global seed so that my work is reproducible. Should I use numpy.random.seed or random.seed? From the link in the comments, I understand that they are different, and that the numpy version is not thread-safe. I want to know specifically which one to use to create IPython notebooks for data analysis. Some of the algorithms from scikit-learn involve generating random numbers, and I want to be sure that the notebook shows the same results on every run.

Response: Should I use np.random.seed or random.seed? That depends on whether in your code you are using numpy's random number generator or the one in random. The random number generators in numpy.random and random have totally separate internal states, so numpy.random.seed() will not affect the random sequences produced by random.random(), and likewise random.seed() will not affect numpy.random.randn() etc. If you are using both random and numpy.random in your code then you will need to separately set the seeds for both. Update Your question seems to be specifically about scikit-learn's random number generators. As far as I can tell, scikit-learn uses numpy.random throughout, so you should use np.random.seed() rather than random.seed(). One important caveat is that np.random is not threadsafe - if you set a global seed, then launch several subprocesses and generate random numbers within them using np.random, each subprocess will inherit the RNG state from its parent, meaning that you will get identical random variates in each subprocess. The usual way around this problem is to pass a different seed (or numpy.random.Random instance) to each subprocess, such that each one has a separate local RNG state. Since some parts of scikit-learn can run in parallel using joblib, you will see that some classes and functions have an option to pass either a seed or an np.random.RandomState instance (e.g. the random_state= parameter to sklearn.decomposition.MiniBatchSparsePCA). I tend to use a single global seed for a script, then generate new random seeds based on the global seed for any parallel functions.


Question:
How to get most informative features for scikit-learn classifiers? The classifiers in machine learning packages like liblinear and nltk offer a method show_most_informative_features(), which is really helpful for debugging features:
```
viagra = None          ok : spam     =      4.5 : 1.0
hello = True           ok : spam     =      4.5 : 1.0
hello = None           spam : ok     =      3.3 : 1.0
viagra = True          spam : ok     =      3.3 : 1.0
casino = True          spam : ok     =      2.0 : 1.0
casino = None          ok : spam     =      1.5 : 1.0
```
My question is if something similar is implemented for the classifiers in scikit-learn. I searched the documentation, but couldn't find anything the like. If there is no such function yet, does somebody know a workaround how to get to those values?

Response:
The classifiers themselves do not record feature names, they just see numeric arrays. However, if you extracted your features using a Vectorizer/CountVectorizer/TfidfVectorizer/DictVectorizer, and you are using a linear model (e.g. LinearSVC or Naive Bayes) then you can apply the same trick that the document classification example uses. Example (untested, may contain a bug or two):
```
def print_top10(vectorizer, clf, class_labels):
    """Prints features with the highest coefficient values, per class"""
    feature_names = vectorizer.get_feature_names()
    for i, class_label in enumerate(class_labels):
        top10 = np.argsort(clf.coef_[i])[-10:]
        print("%s: %s" % (class_label,
              " ".join(feature_names[j] for j in top10)))
```
This is for multiclass classification; for the binary case, I think you should use clf.coef_[0] only. You may have to sort the class_labels.


Question:
Convert numpy array type and values from Float64 to Float32 I am trying to convert threshold array(pickle file of isolation forest from scikit learn) of type from Float64 to Float32
```
for i in range(len(tree.tree_.threshold)):
    tree.tree_.threshold[i] = tree.tree_.threshold[i].astype(np.float32)
```
 Then Printing it
```
for value in tree.tree_.threshold[:5]:
    print(type(value))
    print(value)
```
the output i am getting is :
```
<class 'numpy.float64'>
526226.0
<class 'numpy.float64'>
91.9514312744
<class 'numpy.float64'>
3.60330319405
<class 'numpy.float64'>
-2.0
<class 'numpy.float64'>
-2.0
```
I am not getting a proper conversion to Float32. I want to convert values and their type to Float32, Did anybody have any workaround this ?

Response:
The problem is that you do not do any type conversion of the numpy array. You calculate a float32 variable and put it as an entry into a float64 numpy array. numpy then converts it properly back to float64 Try someting like this:
```
a = np.zeros(4,dtype="float64")
print a.dtype
print type(a[0])
a = np.float32(a)
print a.dtype
print type(a[0])
```
The output (tested with python 2.7)
```
float64
<type 'numpy.float64'>
float32
<type 'numpy.float32'>
```
a is in your case the array tree.tree_.threshold


Question: Differences in SciKit Learn, Keras, or Pytorch [closed] Closed. This question is opinion-based. It is not currently accepting answers. Want to improve this question? Because this question may lead to opinionated discussion, debate, and answers, it has been closed. You may edit the question if you feel you can improve it so that it requires answers that include facts and citations or a detailed explanation of the proposed solution. If edited, the question will be reviewed and might be reopened. Closed 6 years ago. The community reviewed whether to reopen this question 1 year ago and left it closed: Needs more focus Guide the asker to update the question so it focuses on a single, specific problem. Narrowing the question will help others answer the question concisely. You may edit the question if you feel you can improve it yourself. If edited, the question will be reviewed and might be reopened. Improve this question Are these libraries fairly interchangeable? Looking here, https://stackshare.io/stackups/keras-vs-pytorch-vs-scikit-learn, it seems the major difference is the underlying framework (at least for PyTorch).

Response: Yes, there is a major difference. SciKit Learn is a general machine learning library, built on top of NumPy. It features a lot of machine learning algorithms such as support vector machines, random forests, as well as a lot of utilities for general pre- and postprocessing of data. It is not a neural network framework. PyTorch is a deep learning framework, consisting of A vectorized math library similar to NumPy, but with GPU support and a lot of neural network related operations (such as softmax or various kinds of activations) Autograd - an algorithm which can automatically calculate gradients of your functions, defined in terms of the basic operations Gradient-based optimization routines for large scale optimization, dedicated to neural network optimization Neural-network related utility functions Keras is a higher-level deep learning framework, which abstracts many details away, making code simpler and more concise than in PyTorch or TensorFlow, at the cost of limited hackability. It abstracts away the computation backend, which can be TensorFlow, Theano or CNTK. It does not support a PyTorch backend, but that's not something unfathomable - you can consider it a simplified and streamlined subset of the above. In short, if you are going with "classic", non-neural algorithms, neither PyTorch nor Keras will be useful for you. If you're doing deep learning, scikit-learn may still be useful for its utility part; aside from it you will need the actual deep learning framework, where you can choose between Keras and PyTorch but you're unlikely to use both at the same time. This is very subjective, but in my view, if you're working on a novel algorithm, you're more likely to go with PyTorch (or TensorFlow or some other lower-level framework) for flexibility. If you're adapting a known and tested algorithm to a new problem setting, you may want to go with Keras for its greater simplicity and lower entry level.


Question:
transform scipy sparse csr to pandas? I have used the
```
sklearn.preprocessing.OneHotEncoder
```
to transform some data the output is scipy.sparse.csr.csr_matrix how can I merge it back into my original dataframe along with the other columns? I tried to use pd.concat but I get
```
TypeError: cannot concatenate a non-NDFrame object
```
Thanks

Response:
If A is csr_matrix, you can use .toarray() (there's also .todense() that produces a numpy matrix, which is also works for the DataFrame constructor):
```
df = pd.DataFrame(A.toarray())
```
You can then use this with pd.concat().
```
A = csr_matrix([[1, 0, 2], [0, 3, 0]])

  (0, 0)    1
  (0, 2)    2
  (1, 1)    3

<class 'scipy.sparse.csr.csr_matrix'>

pd.DataFrame(A.todense())

   0  1  2
0  1  0  2
1  0  3  0

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2 entries, 0 to 1
Data columns (total 3 columns):
0    2 non-null int64
1    2 non-null int64
2    2 non-null int64
```
In version 0.20, pandas introduced sparse data structures, including the SparseDataFrame. In pandas 1.0, SparseDataFrame was removed: In older versions of pandas, the SparseSeries and SparseDataFrame classes were the preferred way to work with sparse data. With the advent of extension arrays, these subclasses are no longer needed. Their purpose is better served by using a regular Series or DataFrame with sparse values instead. The migration guide shows how to use these new data structures. For instance, to create a DataFrame from a sparse matrix:
```
from scipy.sparse import csr_matrix

A = csr_matrix([[1, 0, 2], [0, 3, 0]])

df = pd.DataFrame.sparse.from_spmatrix(A, columns=['A', 'B', 'C'])

df

   A  B  C
0  1  0  2
1  0  3  0

df.dtypes
A    Sparse[float64, 0]
B    Sparse[float64, 0]
C    Sparse[float64, 0]
dtype: object
```
Alternatively, you can pass sparse matrices to sklearn to avoid running out of memory when converting back to pandas. Just convert your other data to sparse format by passing a numpy array to the scipy.sparse.csr_matrix constructor and use scipy.sparse.hstack to combine (see docs).


Question: PCA For categorical features? In my understanding, I thought PCA can be performed only for continuous features. But while trying to understand the difference between onehot encoding and label encoding came through a post in the following link: When to use One Hot Encoding vs LabelEncoder vs DictVectorizor? It states that one hot encoding followed by PCA is a very good method, which basically means PCA is applied for categorical features. Hence confused, please suggest me on the same.

Response: I disagree with the others. While you can use PCA on binary data (e.g. one-hot encoded data) that does not mean it is a good thing, or it will work very well. PCA is designed for continuous variables. It tries to minimize variance (=squared deviations). The concept of squared deviations breaks down when you have binary variables. So yes, you can use PCA. And yes, you get an output. It even is a least-squared output: it's not as if PCA would segfault on such data. It works, but it is just much less meaningful than you'd want it to be; and supposedly less meaningful than e.g. frequent pattern mining.


Question: What does `sample_weight` do to the way a `DecisionTreeClassifier` works in sklearn? I've read from the relevant documentation that : Class balancing can be done by sampling an equal number of samples from each class, or preferably by normalizing the sum of the sample weights (sample_weight) for each class to the same value. But, it is still unclear to me how this works. If I set sample_weight with an array of only two possible values, 1's and 2's, does this mean that the samples with 2's will get sampled twice as often as the samples with 1's when doing the bagging? I cannot think of a practical example for this.

Response:
Some quick preliminaries: Let's say we have a classification problem with K classes. In a region of feature space represented by the node of a decision tree, recall that the "impurity" of the region is measured by quantifying the inhomogeneity, using the probability of the class in that region. Normally, we estimate:
```
Pr(Class=k) = #(examples of class k in region) / #(total examples in region)
```
The impurity measure takes as input, the array of class probabilities:
```
[Pr(Class=1), Pr(Class=2), ..., Pr(Class=K)]
```
and spits out a number, which tells you how "impure" or how inhomogeneous-by-class the region of feature space is. For example, the gini measure for a two class problem is 2*p*(1-p), where p = Pr(Class=1) and 1-p=Pr(Class=2). Now, basically the short answer to your question is: sample_weight augments the probability estimates in the probability array ... which augments the impurity measure ... which augments how nodes are split ... which augments how the tree is built ... which augments how feature space is diced up for classification. I believe this is best illustrated through example. First consider the following 2-class problem where the inputs are 1 dimensional:
```
from sklearn.tree import DecisionTreeClassifier as DTC

X = [[0],[1],[2]] # 3 simple training examples
Y = [ 1,  2,  1 ] # class labels

dtc = DTC(max_depth=1)
```
So, we'll look trees with just a root node and two children. Note that the default impurity measure the gini measure. Case 1: no sample_weight
```
dtc.fit(X,Y)
print dtc.tree_.threshold
# [0.5, -2, -2]
print dtc.tree_.impurity
# [0.44444444, 0, 0.5]
```
The first value in the threshold array tells us that the 1st training example is sent to the left child node, and the 2nd and 3rd training examples are sent to the right child node. The last two values in threshold are placeholders and are to be ignored. The impurity array tells us the computed impurity values in the parent, left, and right nodes respectively. In the parent node, p = Pr(Class=1) = 2. / 3., so that gini = 2*(2.0/3.0)*(1.0/3.0) = 0.444..... You can confirm the child node impurities as well. Case 2: with sample_weight Now, let's try:
```
dtc.fit(X,Y,sample_weight=[1,2,3])
print dtc.tree_.threshold
# [1.5, -2, -2]
print dtc.tree_.impurity
# [0.44444444, 0.44444444, 0.]
```
You can see the feature threshold is different. sample_weight also affects the impurity measure in each node. Specifically, in the probability estimates, the first training example is counted the same, the second is counted double, and the third is counted triple, due to the sample weights we've provided. The impurity in the parent node region is the same. This is just a coincidence. We can compute it directly:
```
p = Pr(Class=1) = (1+3) / (1+2+3) = 2.0/3.0
```
The gini measure of 4/9 follows. Now, you can see from the chosen threshold that the first and second training examples are sent to the left child node, while the third is sent to the right. We see that impurity is calculated to be 4/9 also in the left child node because:
```
p = Pr(Class=1) = 1 / (1+2) = 1/3.
```
The impurity of zero in the right child is due to only one training example lying in that region. You can extend this with non-integer sample-wights similarly. I recommend trying something like sample_weight = [1,2,2.5], and confirming the computed impurities.


Question:
Scikit Learn - K-Means - Elbow - criterion Today i'm trying to learn something about K-means. I Have understand the algorithm and i know how it works. Now i'm looking for the right k... I found the elbow criterion as a method to detect the right k but i do not understand how to use it with scikit learn?! In scikit learn i'm clustering things in this way
```
kmeans = KMeans(init='k-means++', n_clusters=n_clusters, n_init=10)
kmeans.fit(data)
```
So should i do this several times for n_clusters = 1...n and watch at the Error rate to get the right k ? think this would be stupid and would take a lot of time?!

Response:
If the true label is not known in advance(as in your case), then K-Means clustering can be evaluated using either Elbow Criterion or Silhouette Coefficient. Elbow Criterion Method: The idea behind elbow method is to run k-means clustering on a given dataset for a range of values of k (num_clusters, e.g k=1 to 10), and for each value of k, calculate sum of squared errors (SSE). After that, plot a line graph of the SSE for each value of k. If the line graph looks like an arm - a red circle in below line graph (like angle), the "elbow" on the arm is the value of optimal k (number of cluster). Here, we want to minimize SSE. SSE tends to decrease toward 0 as we increase k (and SSE is 0 when k is equal to the number of data points in the dataset, because then each data point is its own cluster, and there is no error between it and the center of its cluster). So the goal is to choose a small value of k that still has a low SSE, and the elbow usually represents where we start to have diminishing returns by increasing k. Let's consider iris datasets,
```
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt

iris = load_iris()
X = pd.DataFrame(iris.data, columns=iris['feature_names'])
#print(X)
data = X[['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)']]

sse = {}
for k in range(1, 10):
    kmeans = KMeans(n_clusters=k, max_iter=1000).fit(data)
    data["clusters"] = kmeans.labels_
    #print(data["clusters"])
    sse[k] = kmeans.inertia_ # Inertia: Sum of distances of samples to their closest cluster center
plt.figure()
plt.plot(list(sse.keys()), list(sse.values()))
plt.xlabel("Number of cluster")
plt.ylabel("SSE")
plt.show()
```
Plot for above code: We can see in plot, 3 is the optimal number of clusters (encircled red) for iris dataset, which is indeed correct. Silhouette Coefficient Method: From sklearn documentation, A higher Silhouette Coefficient score relates to a model with better-defined clusters. The Silhouette Coefficient is defined for each sample and is composed of two scores: ` a: The mean distance between a sample and all other points in the same class. b: The mean distance between a sample and all other points in the next nearest cluster. The Silhouette Coefficient is for a single sample is then given as: Now, to find the optimal value of k for KMeans, loop through 1..n for n_clusters in KMeans and calculate Silhouette Coefficient for each sample. A higher Silhouette Coefficient indicates that the object is well matched to its own cluster and poorly matched to neighboring clusters.
```
from sklearn.metrics import silhouette_score
from sklearn.datasets import load_iris
from sklearn.cluster import KMeans

X = load_iris().data
y = load_iris().target

for n_cluster in range(2, 11):
    kmeans = KMeans(n_clusters=n_cluster).fit(X)
    label = kmeans.labels_
    sil_coeff = silhouette_score(X, label, metric='euclidean')
    print("For n_clusters={}, The Silhouette Coefficient is {}".format(n_cluster, sil_coeff))
```
Output - For n_clusters=2, The Silhouette Coefficient is 0.680813620271 For n_clusters=3, The Silhouette Coefficient is 0.552591944521 For n_clusters=4, The Silhouette Coefficient is 0.496992849949 For n_clusters=5, The Silhouette Coefficient is 0.488517550854 For n_clusters=6, The Silhouette Coefficient is 0.370380309351 For n_clusters=7, The Silhouette Coefficient is 0.356303270516 For n_clusters=8, The Silhouette Coefficient is 0.365164535737 For n_clusters=9, The Silhouette Coefficient is 0.346583642095 For n_clusters=10, The Silhouette Coefficient is 0.328266088778 As we can see, n_clusters=2 has highest Silhouette Coefficient. This means that 2 should be the optimal number of cluster, Right? But here's the catch. Iris dataset has 3 species of flower, which contradicts the 2 as an optimal number of cluster. So despite n_clusters=2 having highest Silhouette Coefficient, We would consider n_clusters=3 as optimal number of cluster due to - Iris dataset has 3 species. (Most Important) n_clusters=3 has the 2nd highest value of Silhouette Coefficient. So choosing n_clusters=3 is the optimal no. of cluster for iris dataset. Choosing optimal no. of the cluster will depend on the type of datasets and the problem we are trying to solve. But most of the cases, taking highest Silhouette Coefficient will yield an optimal number of cluster. Hope it helps!


Question: Why is scikit-learn SVM.SVC() extremely slow? I tried to use SVM classifier to train a data with about 100k samples, but I found it to be extremely slow and even after two hours there was no response. When the dataset has around 1k samples, I can get the result immediately. I also tried SGDClassifier and naïve bayes which is quite fast and I got results within couple of minutes. Could you explain this phenomena?

Response:
General remarks about SVM-learning SVM-training with nonlinear-kernels, which is default in sklearn's SVC, is complexity-wise approximately: O(n_samples^2 * n_features) link to some question with this approximation given by one of sklearn's devs. This applies to the SMO-algorithm used within libsvm, which is the core-solver in sklearn for this type of problem. This changes much when no kernels are used and one uses sklearn.svm.LinearSVC (based on liblinear) or sklearn.linear_model.SGDClassifier. So we can do some math to approximate the time-difference between 1k and 100k samples:
```
1k = 1000^2 = 1.000.000 steps = Time X
100k = 100.000^2 = 10.000.000.000 steps = Time X * 10000 !!!
```
This is only an approximation and can be even worse or less worse (e.g. setting cache-size; trading-off memory for speed-gains)! Scikit-learn specific remarks The situation could also be much more complex because of all that nice stuff scikit-learn is doing for us behind the bars. The above is valid for the classic 2-class SVM. If you are by any chance trying to learn some multi-class data; scikit-learn will automatically use OneVsRest or OneVsAll approaches to do this (as the core SVM-algorithm does not support this). Read up scikit-learns docs to understand this part. The same warning applies to generating probabilities: SVM's do not naturally produce probabilities for final-predictions. So to use these (activated by parameter) scikit-learn uses a heavy cross-validation procedure called Platt scaling which will take a lot of time too! Scikit-learn documentation Because sklearn has one of the best docs, there is often a good part within these docs to explain something like that (link):


Question: Determining the most contributing features for SVM classifier in sklearn I have a dataset and I want to train my model on that data. After training, I need to know the features that are major contributors in the classification for a SVM classifier. There is something called feature importance for forest algorithms, is there anything similar?

Response:
Yes, there is attribute coef_ for SVM classifier but it only works for SVM with linear kernel. For other kernels it is not possible because data are transformed by kernel method to another space, which is not related to input space, check the explanation.
```
from matplotlib import pyplot as plt
from sklearn import svm

def f_importances(coef, names):
    imp = coef
    imp,names = zip(*sorted(zip(imp,names)))
    plt.barh(range(len(names)), imp, align='center')
    plt.yticks(range(len(names)), names)
    plt.show()

features_names = ['input1', 'input2']
svm = svm.SVC(kernel='linear')
svm.fit(X, Y)
f_importances(svm.coef_, features_names)
```
And the output of the function looks like this:


Question:
Scikit-learn predict_proba gives wrong answers This is a follow-up question from How to know what classes are represented in return array from predict_proba in Scikit-learn In that question, I quoted the following code:
```

```python
import sklearn
```

```python
sklearn.__version__
#Output
#'0.13.1'
```

```python
from sklearn import svm
```

```python
model = svm.SVC(probability=True)
```

```python
X = [[1,2,3], [2,3,4]] # feature vectors
```

```python
Y = ['apple', 'orange'] # classes
```

```python
model.fit(X, Y)
```

```python
model.predict_proba([1,2,3])
#Output
#array([[ 0.39097541,  0.60902459]])
#```
#I discovered in that question this result represents the probability of the point belonging to each class, in the order given by model.classes_
#```
```

```python
zip(model.classes_, model.predict_proba([1,2,3])[0])
#Output
#[('apple', 0.39097541289393828), ('orange', 0.60902458710606167)]
#```
#So... this answer, if interpreted correctly, says that the point is probably an 'orange' (with a fairly low confidence, due to the tiny amount of data). But intuitively, this result is obviously incorrect, since the point given was identical to the training data for 'apple'. Just to be sure, I tested the reverse as well:
#```
```

```python
zip(model.classes_, model.predict_proba([2,3,4])[0])
#Output
#[('apple', 0.60705475211840931), ('orange', 0.39294524788159074)]
#```
#Again, obviously incorrect, but in the other direction. Finally, I tried it with points that were much further away.
#```
```

```python
X = [[1,1,1], [20,20,20]] # feature vectors
```

```python
model.fit(X, Y)
```

```python
zip(model.classes_, model.predict_proba([1,1,1])[0])
#Output
#[('apple', 0.33333332048410247), ('orange', 0.66666667951589786)]
#```
#Again, the model predicts the wrong probabilities. BUT, the model.predict function gets it right!
#```
```

```python
model.predict([1,1,1])[0]
#Output
#'apple'
#```
#Now, I remember reading something in the docs about predict_proba being inaccurate for small datasets, though I can't seem to find it again. Is this the expected behaviour, or am I doing something wrong? If this IS the expected behaviour, then why does the predict and predict_proba function disagree one the output? And importantly, how big does the dataset need to be before I can trust the results from predict_proba? -------- UPDATE -------- Ok, so I did some more 'experiments' into this: the behaviour of predict_proba is heavily dependent on 'n', but not in any predictable way!
#```
```

```python
def train_test(n):
    X = [[1,2,3], [2,3,4]] * n
    Y = ['apple', 'orange'] * n
    model.fit(X, Y)
    print "n =", n, zip(model.classes_, model.predict_proba([1,2,3])[0])

```

```python
train_test(1)
#Output
#n = 1 [('apple', 0.39097541289393828), ('orange', 0.60902458710606167)]
```

```python
for n in range(1,10):
    train_test(n)

#Output
#n = 1 [('apple', 0.39097541289393828), ('orange', 0.60902458710606167)]
#n = 2 [('apple', 0.98437355278112448), ('orange', 0.015626447218875527)]
#n = 3 [('apple', 0.90235408180319321), ('orange', 0.097645918196806694)]
#n = 4 [('apple', 0.83333299908143665), ('orange', 0.16666700091856332)]
#n = 5 [('apple', 0.85714254878984497), ('orange', 0.14285745121015511)]
#n = 6 [('apple', 0.87499969631893626), ('orange', 0.1250003036810636)]
#n = 7 [('apple', 0.88888844127886335), ('orange', 0.11111155872113669)]
#n = 8 [('apple', 0.89999988018127364), ('orange', 0.10000011981872642)]
#n = 9 [('apple', 0.90909082368682159), ('orange', 0.090909176313178491)]
#```
#How should I use this function safely in my code? At the very least, is there any value of n for which it will be guaranteed to agree with the result of model.predict?
```

Response: predict_probas is using the Platt scaling feature of libsvm to callibrate probabilities, see: How does sklearn.svm.svc's function predict_proba() work internally? So indeed the hyperplane predictions and the proba calibration can disagree, especially if you only have 2 samples in your dataset. It's weird that the internal cross validation done by libsvm for scaling the probabilities does not fail (explicitly) in this case. Maybe this is a bug. One would have to dive into the Platt scaling code of libsvm to understand what's happening.


Question:
How to split data on balanced training set and test set on sklearn I am using sklearn for multi-classification task. I need to split alldata into train_set and test_set. I want to take randomly the same sample number from each class. Actually, I amusing this function
```
X_train, X_test, y_train, y_test = cross_validation.train_test_split(Data, Target, test_size=0.3, random_state=0)
```
but it gives unbalanced dataset! Any suggestion.

Response:
Although Christian's suggestion is correct, technically train_test_split should give you stratified results by using the stratify param. So you could do:
```
X_train, X_test, y_train, y_test = cross_validation.train_test_split(Data, Target, test_size=0.3, random_state=0, stratify=Target)
```
The trick here is that it starts from version 0.17 in sklearn. From the documentation about the parameter stratify: stratify : array-like or None (default is None) If not None, data is split in a stratified fashion, using this as the labels array. New in version 0.17: stratify splitting


Question:
module 'sklearn' has no attribute 'cross_validation' I am trying to split my dataset into training and testing dataset, but I am getting this error:
```
X_train,X_test,Y_train,Y_test = sklearn.cross_validation.train_test_split(X,df1['ENTRIESn_hourly'])
```
```
AttributeError                            Traceback (most recent call last)
<ipython-input-53-5445dab94861> in <module>()
----> 1 X_train,X_test,Y_train,Y_test = sklearn.cross_validation.train_test_split(X,df1['ENTRIESn_hourly'])

AttributeError: module 'sklearn' has no attribute 'cross_validation'
```
How can I handle this?

Response: sklearn does not automatically import its subpackages. If you only imported via: import sklearn, then it won't work. Import with import sklearn.cross_validation instead. Further, sklearn.cross_validation will be deprecated in version 0.20. Use sklearn.model_selection.train_test_split instead.


Question:
sklearn and large datasets I have a dataset of 22 GB. I would like to process it on my laptop. Of course I can't load it in memory. I use a lot sklearn but for much smaller datasets. In this situations the classical approach should be something like. Read only part of the data -> Partial train your estimator -> delete the data -> read other part of the data -> continue to train your estimator. I have seen that some sklearn algorithm have the partial fit method that should allow us to train the estimator with various subsamples of the data. Now I am wondering is there an easy why to do that in sklearn? I am looking for something like
```
r = read_part_of_data('data.csv')
m = sk.my_model
`for i in range(n):
     x = r.read_next_chunk(20 lines)
     m.partial_fit(x)

m.predict(new_x)
```
Maybe sklearn is not the right tool for these kind of things? Let me know.

Response: I think sklearn is fine for larger data. If your chosen algorithms support partial_fit or an online learning approach then you're on track. One thing to be aware of is that your chunk size may influence your success. This link may be useful... Working with big data in python and numpy, not enough ram, how to save partial results on disc? I agree that h5py is useful but you may wish to use tools that are already in your quiver. Another thing you can do is to randomly pick whether or not to keep a row in your csv file...and save the result to a .npy file so it loads quicker. That way you get a sampling of your data that will allow you to start playing with it with all algorithms...and deal with the bigger data issue along the way(or not at all! sometimes a sample with a good approach is good enough depending on what you want).


Question: SKLearn MinMaxScaler - scale specific columns only [duplicate] This question already has answers here: pandas dataframe columns scaling with sklearn (9 answers) Closed 3 years ago. I'd like to scale some (but not all) of the columns in a Pandas dataFrame using a MinMaxScaler. How can I do it?

Response:
Demo:
```

```python
df = pd.DataFrame(np.random.randn(5, 3), index=list('abcde'), columns=list('xyz'))
```

```python
df
#Output
#x         y         z
#a -0.325882 -0.299432 -0.182373
#b -0.833546 -0.472082  1.158938
#c -0.328513 -0.664035  0.789414
#d -0.031630 -1.040802 -1.553518
#e  0.813328  0.076450  0.022122
```

```python
from sklearn.preprocessing import MinMaxScaler
```

```python
mms = MinMaxScaler()
```

```python
df[['x','z']] = mms.fit_transform(df[['x','z']])
```

```python
df
#Output
#x         y         z
#a  0.308259 -0.299432  0.505500
#b  0.000000 -0.472082  1.000000
#c  0.306662 -0.664035  0.863768
#d  0.486932 -1.040802  0.000000
#e  1.000000  0.076450  0.580891
#```
#the same result can be also achieved using sklearn.preprocessing.minmax_scale:
#```
#from sklearn.preprocessing import minmax_scale
#
#df[['x','z']] = minmax_scale(df[['x','z']])
#```
```


Question:
What does clf mean in machine learning? When doing fitting, I always come across code like
```
clf = svm.SVC(kernel='linear', C=1).fit(X_train, y_train)
```
(from http://scikit-learn.org/stable/modules/cross_validation.html#k-fold) What does clf stand for? I googled around but didn't find any clues.

Response: In the scikit-learn tutorial, it's short for classifier.: We call our estimator instance clf, as it is a classifier.


Question:
What's the difference between KFold and ShuffleSplit CV? It seems like KFold generates the same values every time the object is iterated over, while Shuffle Split generates different indices every time. Is this correct? If so, what are the uses for one over the other?
```
cv = cross_validation.KFold(10, n_folds=2,shuffle=True,random_state=None)
cv2 = cross_validation.ShuffleSplit(10,n_iter=2,test_size=0.5)
print(list(iter(cv)))
print(list(iter(cv)))
print(list(iter(cv2)))
print(list(iter(cv2)))
```
Yields the following output:
```
[(array([1, 3, 5, 8, 9]), array([0, 2, 4, 6, 7])), (array([0, 2, 4, 6, 7]), array([1, 3, 5, 8, 9]))]
[(array([1, 3, 5, 8, 9]), array([0, 2, 4, 6, 7])), (array([0, 2, 4, 6, 7]), array([1, 3, 5, 8, 9]))]
[(array([4, 6, 3, 2, 7]), array([8, 1, 9, 0, 5])), (array([3, 6, 7, 0, 5]), array([9, 1, 8, 4, 2]))]
[(array([3, 0, 2, 1, 7]), array([5, 6, 9, 4, 8])), (array([0, 7, 1, 3, 8]), array([6, 2, 5, 4, 9]))]
```

Response: Difference in KFold and ShuffleSplit output KFold will divide your data set into prespecified number of folds, and every sample must be in one and only one fold. A fold is a subset of your dataset. ShuffleSplit will randomly sample your entire dataset during each iteration to generate a training set and a test set. The test_size and train_size parameters control how large the test and training test set should be for each iteration. Since you are sampling from the entire dataset during each iteration, values selected during one iteration, could be selected again during another iteration. Summary: ShuffleSplit works iteratively, KFold just divides the dataset into k folds. Difference when doing validation In KFold, during each round you will use one fold as the test set and all the remaining folds as your training set. However, in ShuffleSplit, during each round n you should only use the training and test set from iteration n. As your data set grows, cross validation time increases, making shufflesplits a more attractive alternate. If you can train your algorithm, with a certain percentage of your data as opposed to using all k-1 folds, ShuffleSplit is an attractive option.


Question: Python scikit learn pca.explained_variance_ratio_ cutoff When choosing the number of principal components (k), we choose k to be the smallest value so that for example, 99% of variance, is retained. However, in the Python Scikit learn, I am not 100% sure pca.explained_variance_ratio_ = 0.99 is equal to "99% of variance is retained"? Could anyone enlighten? Thanks. The Python Scikit learn PCA manual is here http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html#sklearn.decomposition.PCA

Response:
Yes, you are nearly right. The pca.explained_variance_ratio_ parameter returns a vector of the variance explained by each dimension. Thus pca.explained_variance_ratio_[i] gives the variance explained solely by the i+1st dimension. You probably want to do pca.explained_variance_ratio_.cumsum(). That will return a vector x such that x[i] returns the cumulative variance explained by the first i+1 dimensions.
```
import numpy as np
from sklearn.decomposition import PCA

np.random.seed(0)
my_matrix = np.random.randn(20, 5)

my_model = PCA(n_components=5)
my_model.fit_transform(my_matrix)

print my_model.explained_variance_
print my_model.explained_variance_ratio_
print my_model.explained_variance_ratio_.cumsum()
```
```
[ 1.50756565  1.29374452  0.97042041  0.61712667  0.31529082]
[ 0.32047581  0.27502207  0.20629036  0.13118776  0.067024  ]
[ 0.32047581  0.59549787  0.80178824  0.932976    1.        ]
```
So in my random toy data, if I picked k=4 I would retain 93.3% of the variance.


Question: Controlling the threshold in Logistic Regression in Scikit Learn I am using the LogisticRegression() method in scikit-learn on a highly unbalanced data set. I have even turned the class_weight feature to auto. I know that in Logistic Regression it should be possible to know what is the threshold value for a particular pair of classes. Is it possible to know what the threshold value is in each of the One-vs-All classes the LogisticRegression() method designs? I did not find anything in the documentation page. Does it by default apply the 0.5 value as threshold for all the classes regardless of the parameter values?

Response:
There is a little trick that I use, instead of using model.predict(test_data) use model.predict_proba(test_data). Then use a range of values for thresholds to analyze the effects on the prediction;
```
pred_proba_df = pd.DataFrame(model.predict_proba(x_test))
threshold_list = [0.05,0.1,0.15,0.2,0.25,0.3,0.35,0.4,0.45,0.5,0.55,0.6,0.65,.7,.75,.8,.85,.9,.95,.99]
for i in threshold_list:
    print ('\n******** For i = {} ******'.format(i))
    Y_test_pred = pred_proba_df.applymap(lambda x: 1 if x>i else 0)
    test_accuracy = metrics.accuracy_score(Y_test.values.reshape(Y_test.values.size,1),
                                           Y_test_pred.iloc[:,1].values.reshape(Y_test_pred.iloc[:,1].values.size,1))
    print('Our testing accuracy is {}'.format(test_accuracy))

    print(confusion_matrix(Y_test.values.reshape(Y_test.values.size,1),
                           Y_test_pred.iloc[:,1].values.reshape(Y_test_pred.iloc[:,1].values.size,1)))
```
Best!