Question: using sklearn.train_test_split for Imbalanced data I have a very imbalanced dataset. I used sklearn.train_test_split function to extract the train dataset. Now I want to oversample the train dataset, so I used to count number of type1(my data set has 2 categories and types(type1 and tupe2) but approximately all of my train data are type1. So I cant oversample. Previously I used to split train test datasets with my written code. In that code 0.8 of all type1 data and 0.8 of all type2 data were in the train dataset. How I can use this method with train_test_split function or other spliting methods in sklearn? *I should just use sklearn or my own written methods.

Response:
You're looking for stratification. Why? There's a parameter stratify in method train_test_split to which you can give the labels list e.g. :
```
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    stratify=y,
                                                    test_size=0.2)
```
There's also StratifiedShuffleSplit.


Question: How to balance classification using DecisionTreeClassifier? I have a data set where the classes are unbalanced. The classes are either 0, 1 or 2. How can I calculate the prediction error for each class and then re-balance weights accordingly in scikit-learn?

Response: If you want to fully balance (treat each class as equally important) you can simply pass class_weight='balanced', as it is stated in the docs: The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))


Question:
How to one-hot-encode from a pandas column containing a list? I would like to break down a pandas column consisting of a list of elements into as many columns as there are unique elements i.e. one-hot-encode them (with value 1 representing a given element existing in a row and 0 in the case of absence). For example, taking dataframe df
```
Col1   Col2         Col3
 C      33     [Apple, Orange, Banana]
 A      2.5    [Apple, Grape]
 B      42     [Banana]
```
I would like to convert this to: df
```
Col1   Col2   Apple   Orange   Banana   Grape
 C      33     1        1        1       0
 A      2.5    1        0        0       1
 B      42     0        0        1       0
```
How can I use pandas/sklearn to achieve this?

Response:
We can also use sklearn.preprocessing.MultiLabelBinarizer: Often we want to use sparse DataFrame for the real world data in order to save a lot of RAM. Sparse solution (for Pandas v0.25.0+)
```
from sklearn.preprocessing import MultiLabelBinarizer

mlb = MultiLabelBinarizer(sparse_output=True)

df = df.join(
            pd.DataFrame.sparse.from_spmatrix(
                mlb.fit_transform(df.pop('Col3')),
                index=df.index,
                columns=mlb.classes_))
```
result:
```

```python
df
#Output
#Col1  Col2  Apple  Banana  Grape  Orange
#0    C  33.0      1       1      0       1
#1    A   2.5      1       0      1       0
#2    B  42.0      0       1      0       0
```

```python
df.dtypes
#Output
#Col1                object
#Col2               float64
#Apple     Sparse[int32, 0]
#Banana    Sparse[int32, 0]
#Grape     Sparse[int32, 0]
#Orange    Sparse[int32, 0]
#dtype: object
```

```python
df.memory_usage()
#Output
#Index     128
#Col1       24
#Col2       24
#Apple      16    #  <--- NOTE!
#Banana     16    #  <--- NOTE!
#Grape       8    #  <--- NOTE!
#Orange      8    #  <--- NOTE!
#dtype: int64
#```
#Dense solution
#```
#mlb = MultiLabelBinarizer()
#df = df.join(pd.DataFrame(mlb.fit_transform(df.pop('Col3')),
#                          columns=mlb.classes_,
#                          index=df.index))
#```
#Result:
#```
```

```python
df
#Output
#Col1  Col2  Apple  Banana  Grape  Orange
#0    C  33.0      1       1      0       1
#1    A   2.5      1       0      1       0
#2    B  42.0      0       1      0       0
#```
```


Question:
Arrays used as indices must be of integer (or boolean) type Errors are like this:
```
Traceback (most recent call last):
  File "NearestCentroid.py", line 53, in <module>
    clf.fit(X_train.todense(),y_train)
  File "/usr/local/lib/python2.7/dist-packages/scikit_learn-0.13.1-py2.7-linux-i686.egg/sklearn/neighbors/nearest_centroid.py", line 115, in fit
    variance = np.array(np.power(X - self.centroids_[y], 2))
IndexError: arrays used as indices must be of integer (or boolean) type
```
Codes are like this:
```
distancemetric=['euclidean','l2']
for mtrc in distancemetric:
for shrkthrshld in [None]:
#shrkthrshld=0
#while (shrkthrshld <=1.0):
    clf = NearestCentroid(metric=mtrc,shrink_threshold=shrkthrshld)
    clf.fit(X_train.todense(),y_train)
    y_predicted = clf.predict(X_test.todense())
```
I am using scikit-learn package, X-train, y_train are in LIBSVM format, X is the feature:value pair, y_train is the target/label, X_train is in CSR matric format, the shrink_threshold does not support CSR sparse matrix, so I add .todense() to X_train, then I got this error, could anyone help me fix this? Thanks a lot!

Response: I had a similar problem using the Pystruct pystruct.learners.OneSlackSSVM. It occured because my training labels were floats, in stead of integers. In my case, it was because I initialized the labels with np.ones, without specifying dtype=np.int8. Hope it helps.


Question:
How to get decision function in randomforest in sklearn I am using the following code to get the optimised parameters for randomforest using gridsearchcv.
```
x_train, x_test, y_train, y_test = train_test_split(X, y, random_state=0)
rfc = RandomForestClassifier(random_state=42, class_weight = 'balanced')
param_grid = {
    'n_estimators': [200, 500],
    'max_features': ['auto', 'sqrt', 'log2'],
    'max_depth' : [4,5,6,7,8],
    'criterion' :['gini', 'entropy']
}
k_fold = StratifiedKFold(n_splits=10, shuffle=True, random_state=0)
CV_rfc = GridSearchCV(estimator=rfc, param_grid=param_grid, cv= 10, scoring = 'roc_auc')
CV_rfc.fit(x_train, y_train)
print(CV_rfc.best_params_)
print(CV_rfc.best_score_)
```
Now, I want to apply the tuned parameters to X_test. For that I did the following,
```
pred = CV_rfc.decision_function(x_test)
print(roc_auc_score(y_test, pred))
```
However, decision_function does not seem to support randomforest as I got the following error. AttributeError: 'RandomForestClassifier' object has no attribute 'decision_function'. Is there any other way of doing this? I am happy to provide more details if needed.

Response:
If your intention is to get a model scoring function so that the scoring can be used for auc_roc_score, then you can go for predict_proba()
```
y_pred_proba = CV_rfc.predict_proba(x_test)
print(roc_auc_score(y_test, y_pred_proba[:,1]))
```


Question: What is python's equivalent of R's NA? What is python's equivalent of R's NA? To be more specific: R has NaN, NA, NULL, Inf and -Inf. NA is generally used when there is missing data. What is python's equivalent? How libraries such as numpy and pandas handle missing values? How does scikit-learn handle missing values? Is it different for python 2.7 and python 3?

Response:
nan in numpy is handled well with many functions:
```

```python
import numpy as np
```

```python
a = [1, np.nan, 2, 3]
```

```python
np.nanmean(a)
#Output
#2.0
```

```python
np.nansum(a)
#Output
#6.0
```

```python
np.isnan(a)
#Output
#array([False,  True, False, False], dtype=bool)
#```
```


Question: Macro VS Micro VS Weighted VS Samples F1 Score In sklearn.metrics.f1_score, the f1 score has a parameter called "average". What does macro, micro, weighted, and samples mean? Please elaborate, because in the documentation, it was not explained properly. Or simply answer the following: Why is "samples" best parameter for multilabel classification? Why is micro best for an imbalanced dataset? what's the difference between weighted and macro?

Response: The question is about the meaning of the average parameter in sklearn.metrics.f1_score. As you can see from the code: average=micro says the function to compute f1 by considering total true positives, false negatives and false positives (no matter of the prediction for each label in the dataset) average=macro says the function to compute f1 for each label, and returns the average without considering the proportion for each label in the dataset. average=weighted says the function to compute f1 for each label, and returns the average considering the proportion for each label in the dataset. average=samples says the function to compute f1 for each instance, and returns the average. Use it for multilabel classification.


Question: How to do LabelEncoding or categorical value in Apache Spark I am having dataset contains String columns . How can I encode the string based columns like the one we do in scikit-learn LabelEncoder

Response:
StringIndexer is what you need https://spark.apache.org/docs/1.5.1/ml-features.html#stringindexer
```
from pyspark.ml.feature import StringIndexer

df = sqlContext.createDataFrame(
            [(0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c")],
            ["id", "category"])
indexer = StringIndexer(inputCol="category", outputCol="categoryIndex")
indexed = indexer.fit(df).transform(df)
indexed.show()
```


Question:
Print Estimator Name in SkLearn In Sklearn, is there a way to print out an estimator's class name? I have tried to use the name attribute but that is not working.
```
from sklearn.linear_model import LogisticRegression

def print_estimator_name(estimator):
    print(estimator.__name__)

#Expected Outcome:
print_estimator_name(LogisticRegression())
```
I would expect this to print out the classifier name as above

Response:
I think you are looking for estimator.__class__.__name__ i.e.:
```
from sklearn.linear_model import LogisticRegression

def print_estimator_name(estimator):
    print(estimator.__class__.__name__)

#Expected Outcome:
print_estimator_name(LogisticRegression())
```


Question:
how to set threshold to scikit learn random forest model After seeing the precision_recall_curve, if I want to set threshold = 0.4, how to implement 0.4 into my random forest model (binary classification), for any probability =0.4, label it as 1.
```
from sklearn.ensemble import RandomForestClassifier
  random_forest = RandomForestClassifier(n_estimators=100, oob_score=True, random_state=12)
  random_forest.fit(X_train, y_train)
from sklearn.metrics import accuracy_score
  predicted = random_forest.predict(X_test)
accuracy = accuracy_score(y_test, predicted)
```
Documentation Precision recall

Response:
Assuming you are doing binary classification, it's quite easy:
```
threshold = 0.4

predicted_proba = random_forest.predict_proba(X_test)
predicted = (predicted_proba [:,1] >= threshold).astype('int')

accuracy = accuracy_score(y_test, predicted)
```


Question:
confusion matrix error "Classification metrics can't handle a mix of multilabel-indicator and multiclass targets" I am getting a
```
Classification metrics can't handle a mix of multilabel-indicator and multiclass targets
```
error when I try to use confusion matrix. I am doing my first deep learning project. I am new to it. I am using the mnist dataset provided by keras. I have trained and tested my model successfully. However, when I try to use the scikit learn confusion matrix I get the error stated above. I have searched for an answer and while there are answers on this error, none of them worked for me. From what I found online it probably has something to do with the loss function (I use the categorical_crossentropy in my code). I tried changing it to sparse_categorical_crossentropy but that just gave me the
```
Error when checking target: expected dense_2 to have shape (1,) but got array with shape (10,)
```
when I run the fit() function on the model. This is the code. (I have left out the imports for the sake of brevity)
```
model = Sequential()
model.add(Dense(512, activation='relu', input_shape=(28 * 28,)))
model.add(Dense(10, activation='softmax'))

model.compile(optimizer='Adam', loss='categorical_crossentropy', metrics=['accuracy'])

(train_images, train_labels), (test_images, test_labels) = mnist.load_data()

train_images = train_images.reshape((60000, 28 * 28))
train_images = train_images.astype('float32') / 255

test_images = test_images.reshape((10000, 28 * 28))
test_images = test_images.astype('float32') / 255

train_labels = to_categorical(train_labels)
test_labels = to_categorical(test_labels)

model.fit(train_images, train_labels, epochs=10, batch_size=128)

rounded_predictions = model.predict_classes(test_images, batch_size=128, verbose=0)

cm = confusion_matrix(test_labels, rounded_predictions)
```
How can i fix this?

Response:
Confusion matrix needs both labels & predictions as single-digits, not as one-hot encoded vectors; although you have done this with your predictions using model.predict_classes(), i.e.
```python
rounded_predictions = model.predict_classes(test_images, batch_size=128, verbose=0)
rounded_predictions[1]
# 2
```
your test_labels are still one-hot encoded:
```python
test_labels[1]
# array([0., 0., 1., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)
```
So, you should convert them too to single-digit ones, as follows:
```python
import numpy as np
rounded_labels=np.argmax(test_labels, axis=1)
rounded_labels[1]
# 2
```
After which, the confusion matrix should come up OK:
```python
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(rounded_labels, rounded_predictions)
cm
# result:
array([[ 971,    0,    0,    2,    1,    0,    2,    1,    3,    0],
       [   0, 1121,    2,    1,    0,    1,    3,    0,    7,    0],
       [   5,    4,  990,    7,    5,    3,    2,    7,    9,    0],
       [   0,    0,    0,  992,    0,    2,    0,    7,    7,    2],
       [   2,    0,    2,    0,  956,    0,    3,    3,    2,   14],
       [   3,    0,    0,   10,    1,  872,    3,    0,    1,    2],
       [   5,    3,    1,    1,    9,   10,  926,    0,    3,    0],
       [   0,    7,   10,    1,    0,    2,    0,  997,    1,   10],
       [   5,    0,    3,    7,    5,    7,    3,    4,  937,    3],
       [   5,    5,    0,    9,   10,    3,    0,    8,    3,  966]])
```


Question:
pca.inverse_transform in sklearn after fitting my data into X = my data
```
pca = PCA(n_components=1)
pca.fit(X)
X_pca = pca.fit_transform(X)
```
now X_pca has one dimension. When I perform inverse transformation by definition isn't it supposed to return to original data, that is X, 2-D array? when I do
```
X_ori = pca.inverse_transform(X_pca)
```
I get same dimension however different numbers. Also if I plot both X and X_ori they are different.

Response: When I perform inverse transformation by definition isn't it supposed to return to original data No, you can only expect this if the number of components you specify is the same as the dimensionality of the input data. For any n_components less than this, you will get different numbers than the original dataset after applying the inverse PCA transformation: the following diagrams give an illustration in two dimensions.


Question:
Using explicit (predefined) validation set for grid search with sklearn I have a dataset, which has previously been split into 3 sets: train, validation and test. These sets have to be used as given in order to compare the performance across different algorithms. I would now like to optimize the parameters of my SVM using the validation set. However, I cannot find how to input the validation set explicitly into sklearn.grid_search.GridSearchCV(). Below is some code I've previously used for doing K-fold cross-validation on the training set. However, for this problem I need to use the validation set as given. How can I do that?
```py
from sklearn import svm, cross_validation
from sklearn.grid_search import GridSearchCV

# (some code left out to simplify things)

skf = cross_validation.StratifiedKFold(y_train, n_folds=5, shuffle = True)
clf = GridSearchCV(svm.SVC(tol=0.005, cache_size=6000,
                             class_weight=penalty_weights),
                     param_grid=tuned_parameters,
                     n_jobs=2,
                     pre_dispatch="n_jobs",
                     cv=skf,
                     scoring=scorer)
clf.fit(X_train, y_train)
```

Response:
Use PredefinedSplit
```
ps = PredefinedSplit(test_fold=your_test_fold)
```
then set cv=ps in GridSearchCV test_fold : “array-like, shape (n_samples,) test_fold[i] gives the test set fold of sample i. A value of -1 indicates that the corresponding sample is not part of any test set folds, but will instead always be put into the training fold. Also see here when using a validation set, set the test_fold to 0 for all samples that are part of the validation set, and to -1 for all other samples.


Question:
Predicting missing values with scikit-learn's Imputer module I am writing a very basic program to predict missing values in a dataset using scikit-learn's Imputer class. I have made a NumPy array, created an Imputer object with strategy='mean' and performed fit_transform() on the NumPy array. When I print the array after performing fit_transform(), the 'Nan's remain, and I dont get any prediction. What am I doing wrong here? How do I go about predicting the missing values?
```
import numpy as np
from sklearn.preprocessing import Imputer

X = np.array([[23.56],[53.45],['NaN'],[44.44],[77.78],['NaN'],[234.44],[11.33],[79.87]])

print X

imp = Imputer(missing_values='NaN', strategy='mean', axis=0)
imp.fit_transform(X)

print X
```

Response:
Per the documentation, sklearn.preprocessing.Imputer.fit_transform returns a new array, it doesn't alter the argument array. The minimal fix is therefore:
```
X = imp.fit_transform(X)
```


Question:
Deprecation warnings from sklearn I am using cross_validation from sklearn,
```
from sklearn.cross_validation import train_test_split
```
I get the below warning: cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved.

Response:
Problem: The deprecation warning means that the module is deprecated, i.e. no longer supported. You are using a version for which sklearn.cross_validation is not a module any longer. Solution:
```
from sklearn.model_selection import train_test_split
```
C/O: This post.


Question:
Tensorflow Precision / Recall / F1 score and Confusion matrix I would like to know if there is a way to implement the different score function from the scikit learn package like this one :
```
from sklearn.metrics import confusion_matrix
confusion_matrix(y_true, y_pred)
```
into a tensorflow model to get the different score.
```
with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as sess:
init = tf.initialize_all_variables()
sess.run(init)
for epoch in xrange(1):
        avg_cost = 0.
        total_batch = len(train_arrays) / batch_size
        for batch in range(total_batch):
                train_step.run(feed_dict = {x: train_arrays, y: train_labels})
                avg_cost += sess.run(cost, feed_dict={x: train_arrays, y: train_labels})/total_batch
        if epoch % display_step == 0:
                print "Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(avg_cost)

print "Optimization Finished!"
correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
# Calculate accuracy
accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
print "Accuracy:", batch, accuracy.eval({x: test_arrays, y: test_labels})
```
Will i have to run the session again to get the prediction ?

Response:
You do not really need sklearn to calculate precision/recall/f1 score. You can easily express them in TF-ish way by looking at the formulas: Now if you have your actual and predicted values as vectors of 0/1, you can calculate TP, TN, FP, FN using tf.count_nonzero:
```
TP = tf.count_nonzero(predicted * actual)
TN = tf.count_nonzero((predicted - 1) * (actual - 1))
FP = tf.count_nonzero(predicted * (actual - 1))
FN = tf.count_nonzero((predicted - 1) * actual)
```
Now your metrics are easy to calculate:
```
precision = TP / (TP + FP)
recall = TP / (TP + FN)
f1 = 2 * precision * recall / (precision + recall)
```


Question:
How to generate one hot encoding for DNA sequences? I would like to generate one hot encoding for a set of DNA sequences. For example the sequence ACGTCCA can be represented as below in a transpose manner. But the code below will generate the one hot encoding in horizontal way in which I would prefer it in vertical form. Can anyone help me?
```
ACGTCCA
1000001 - A
0100110 - C
0010000 - G
0001000 - T
```
Example code:
```
from sklearn.preprocessing import OneHotEncoder
import itertools

# two example sequences
seqs = ["ACGTCCA","CGGATTG"]

# split sequences to tokens
tokens_seqs = [seq.split("\\") for seq in seqs]

# convert list of of token-lists to one flat list of tokens
# and then create a dictionary that maps word to id of word,
# like {A: 1, B: 2} here
all_tokens = itertools.chain.from_iterable(tokens_seqs)
word_to_id = {token: idx for idx, token in enumerate(set(all_tokens))}

# convert token lists to token-id lists, e.g. [[1, 2], [2, 2]] here
token_ids = [[word_to_id[token] for token in tokens_seq] for tokens_seq in tokens_seqs]

# convert list of token-id lists to one-hot representation
vec = OneHotEncoder(n_values=len(word_to_id))
X = vec.fit_transform(token_ids)

print X.toarray()
```
However, the code gives me output:
```
[[ 0.  1.]
 [ 1.  0.]]
```
Expected output:
```
[[1. 0. 0. 0. 0. 0. 1. 0. 1. 0. 0. 1. 1. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]
[0. 0. 0. 1. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 1. 1. 0. 0. 0. 1. 0. 0. 0. 0. 1. 1. 0.]]
```

Response:
```
def one_hot_encode(seq):
    mapping = dict(zip("ACGT", range(4)))
    seq2 = [mapping[i] for i in seq]
    return np.eye(4)[seq2]

one_hot_encode("AACGT")

## Output:
array([[1., 0., 0., 0.],
   [1., 0., 0., 0.],
   [0., 1., 0., 0.],
   [0., 0., 1., 0.],
   [0., 0., 0., 1.]])
```


Question:
Will pandas dataframe object work with sklearn kmeans clustering? dataset is pandas dataframe. This is sklearn.cluster.KMeans
```
km = KMeans(n_clusters = n_Clusters)

 km.fit(dataset)

 prediction = km.predict(dataset)
```
This is how I decide which entity belongs to which cluster:
```
for i in range(len(prediction)):
     cluster_fit_dict[dataset.index[i]] = prediction[i]
```
This is how dataset looks:
```
A 1 2 3 4 5 6
 B 2 3 4 5 6 7
 C 1 4 2 7 8 1
 ...
```
where A,B,C are indices Is this the correct way of using k-means?

Response:
Assuming all the values in the dataframe are numeric,
```
# Convert DataFrame to matrix
mat = dataset.values
# Using sklearn
km = sklearn.cluster.KMeans(n_clusters=5)
km.fit(mat)
# Get cluster assignment labels
labels = km.labels_
# Format results as a DataFrame
results = pandas.DataFrame([dataset.index,labels]).T
```
Alternatively, you could try KMeans++ for Pandas.


Question:
Use sklearn TfidfVectorizer with already tokenized inputs? I have a list of tokenized sentences and would like to fit a tfidf Vectorizer. I tried the following:
```
tokenized_list_of_sentences = [['this', 'is', 'one'], ['this', 'is', 'another']]

def identity_tokenizer(text):
  return text

tfidf = TfidfVectorizer(tokenizer=identity_tokenizer, stop_words='english')
tfidf.fit_transform(tokenized_list_of_sentences)
```
which errors out as
```
AttributeError: 'list' object has no attribute 'lower'
```
is there a way to do this? I have a billion sentences and do not want to tokenize them again. They are tokenized before for another stage before this.

Response:
Try initializing the TfidfVectorizer object with the parameter lowercase=False (assuming this is actually desired as you've lowercased your tokens in previous stages).
```py
tokenized_list_of_sentences = [['this', 'is', 'one', 'basketball'], ['this', 'is', 'a', 'football']]

def identity_tokenizer(text):
    return text

tfidf = TfidfVectorizer(tokenizer=identity_tokenizer, stop_words='english', lowercase=False)
tfidf.fit_transform(tokenized_list_of_sentences)
```
Note that I changed the sentences as they apparently only contained stop words which caused another error due to an empty vocabulary.


Question:
Error with Sklearn Random Forest Regressor When trying to fit a Random Forest Regressor model with y data that looks like this:
```
[  0.00000000e+00   1.36094276e+02   4.46608221e+03   8.72660888e+03
   1.31375786e+04   1.73580193e+04   2.29420671e+04   3.12216341e+04
   4.11395711e+04   5.07972062e+04   6.14904935e+04   7.34275322e+04
   7.87333933e+04   8.46302456e+04   9.71074959e+04   1.07146672e+05
   1.17187952e+05   1.26953374e+05   1.37736003e+05   1.47239359e+05
   1.53943242e+05   1.78806710e+05   1.92657725e+05   2.08912711e+05
   2.22855152e+05   2.34532982e+05   2.41391255e+05   2.48699216e+05
   2.62421197e+05   2.79544300e+05   2.95550971e+05   3.13524275e+05
   3.23365158e+05   3.24069067e+05   3.24472999e+05   3.24804951e+05
```
And X data that looks like this:
```
[ 735233.27082176  735234.27082176  735235.27082176  735236.27082176
  735237.27082176  735238.27082176  735239.27082176  735240.27082176
  735241.27082176  735242.27082176  735243.27082176  735244.27082176
  735245.27082176  735246.27082176  735247.27082176  735248.27082176
```
With the following code:
```
regressor = RandomForestRegressor(n_estimators=150, min_samples_split=1)
rgr = regressor.fit(X,y)
```
I get this error:
```
ValueError: Number of labels=600 does not match number of samples=1
```
I assume one of my sets of values is in the wrong format but its not too clear to me from the documentation.

Response:
The shape of X should be [n_samples, n_features], you can transform X by
```
X = X[:, None]
```


Question: what is the difference between 'transform' and 'fit_transform' in sklearn In the sklearn-python toolbox, there are two functions transform and fit_transform about sklearn.decomposition.RandomizedPCA. The description of two functions are as follows But what is the difference between them ?

Response: In scikit-learn estimator api, fit() : used for generating learning model parameters from training data transform() : parameters generated from fit() method,applied upon model to generate transformed data set. fit_transform() : combination of fit() and transform() api on same data set Checkout Chapter-4 from this book & answer from stackexchange for more clarity


Question:
sklearn roc_auc_score with multi_class=="ovr" should have None average available I'm trying to compute the AUC score for a multiclass problem using the sklearn's roc_auc_score() function. I have prediction matrix of shape [n_samples,n_classes] and a ground truth vector of shape [n_samples], named np_pred and np_label respectively. What I'm trying to achieve is the set of AUC scores, one for each classes that I have. To do so I would like to use the average parameter option None and multi_class parameter set to "ovr", but if I run
```
roc_auc_score(y_score=np_pred, y_true=np_label, multi_class="ovr",average=None)
```
I get back
```
ValueError: average must be one of ('macro', 'weighted') for multiclass problems
```
This error is expected from the sklearn function in the case of the multiclass; but if you take a look at the roc_auc_score function source code, you can see that if the multi_class parameter is set to "ovr", and the average is one of the accepted one, the multiClass case is treated as a multiLabel one and the internal multiLabel function accepts None as average parameter. So, by looking at the code, it seems that I should be able to execute a multiclass with a None average in a One vs Rest case but the ifs in the source code do not allow such combination. Am I wrong? In case I'm wrong, from a theoretical point of view should I fake a multilabel case just to have the different AUCs for the different classes or should I write my own function that cycles the different classes and outputs the AUCs? Thanks

Response: According to sklearn documentation, the default parameter for multi_class is 'raised', In documentation it is mentioned, that the default parameter will throw an exception, so you have to mention ovr or ovo explicitly multi_class='ovr'. Refer to the attached screenshot


Question:
Scikit-learn train_test_split with indices How do I get the original indices of the data when using train_test_split()? What I have is the following
```
from sklearn.cross_validation import train_test_split
import numpy as np
data = np.reshape(np.randn(20),(10,2)) # 10 training examples
labels = np.random.randint(2, size=10) # 10 labels
x1, x2, y1, y2 = train_test_split(data, labels, size=0.2)
```
But this does not give the indices of the original data. One workaround is to add the indices to data (e.g. data = [(i, d) for i, d in enumerate(data)]) and then pass them inside train_test_split and then expand again. Are there any cleaner solutions?

Response:
You can use pandas dataframes or series as Julien said but if you want to restrict your-self to numpy you can pass an additional array of indices:
```
from sklearn.model_selection import train_test_split
import numpy as np
n_samples, n_features, n_classes = 10, 2, 2
data = np.random.randn(n_samples, n_features)  # 10 training examples
labels = np.random.randint(n_classes, size=n_samples)  # 10 labels
indices = np.arange(n_samples)
(
    data_train,
    data_test,
    labels_train,
    labels_test,
    indices_train,
    indices_test,
) = train_test_split(data, labels, indices, test_size=0.2)
```


Question: How to set optuna's study.optimize verbosity to 0? I want to set optuna's study.optimize verbosity to 0. I thought optuna.logging.set_verbosity(0) might do it, but I still get the Trial 0 finished with value .... updates for every trial What is the correct way to do this? Unfortunately, extensive searching through the docs still only results in the above method. Many thanks in advance

Response:
Try this:
```
optuna.logging.set_verbosity(optuna.logging.WARNING)
```
It basically changes the level of verbosity. For more level choices, check optuna's official guides here.


Question:
What is the inverse of regularization strength in Logistic Regression? How should it affect my code? I am using sklearn.linear_model.LogisticRegression in scikit learn to run a Logistic Regression.
```none
C : float, optional (default=1.0) Inverse of regularization strength;
    must be a positive float. Like in support vector machines, smaller
    values specify stronger regularization.
```
What does C mean here in simple terms? What is regularization strength?

Response: Regularization is applying a penalty to increasing the magnitude of parameter values in order to reduce overfitting. When you train a model such as a logistic regression model, you are choosing parameters that give you the best fit to the data. This means minimizing the error between what the model predicts for your dependent variable given your data compared to what your dependent variable actually is. The problem comes when you have a lot of parameters (a lot of independent variables) but not too much data. In this case, the model will often tailor the parameter values to idiosyncrasies in your data -- which means it fits your data almost perfectly. However because those idiosyncrasies don't appear in future data you see, your model predicts poorly. To solve this, as well as minimizing the error as already discussed, you add to what is minimized and also minimize a function that penalizes large values of the parameters. Most often the function is λΣθj2, which is some constant λ times the sum of the squared parameter values θj2. The larger λ is the less likely it is that the parameters will be increased in magnitude simply to adjust for small perturbations in the data. In your case however, rather than specifying λ, you specify C=1/λ.


Question: 'verbose' argument in scikit-learn Many scikit-learn functions have a verbose argument that, according to their documentation, "[c]ontrols the verbosity: the higher, the more messages" (e.g., GridSearchCV). Unfortunately, no guidance is provided on which integers are allowed (e.g., can a user set verbosity to 100?) and what level of verbosity corresponds to which integers. I cannot find this information anywhere in the documentation. My question is, which integers map to which levels of verbosity?

Response: Higher integers map to higher verbosity as the docstring says. You can set verbosity=100 but I'm pretty sure it will be the same as verbosity=10. If you are looking for a list of what exactly is printed for each estimator for each integer, you have to look into the source. I think most estimators only have two or three levels of verbosity, I think 3 or above will be the most verbose you can get.


Question:
Why scikit learn confusion matrix is reversed? I have 3 questions: 1) The confusion matrix for sklearn is as follows:
```
TN | FP
FN | TP
```
While when I'm looking at online resources, I find it like this:
```
TP | FP
FN | TN
```
Which one should I consider? 2) Since the above confusion matrix for scikit learn is different than the one I find in other rescources, in a multiclass confusion matrix, what's the structure will be? I'm looking at this post here: Scikit-learn: How to obtain True Positive, True Negative, False Positive and False Negative In that post, @lucidv01d had posted a graph to understand the categories for multiclass. is that category the same in scikit learn? 3) How do you calculate the accuracy of a multiclass? for example, I have this confusion matrix:
```
[[27  6  0 16]
 [ 5 18  0 21]
 [ 1  3  6  9]
 [ 0  0  0 48]]
```
In that same post I referred to in question 2, he has written this equation: Overall accuracy ACC = (TP+TN)/(TP+FP+FN+TN) but isn't that just for binary? I mean, for what class do I replace TP with?

Response:
The reason why sklearn has show their confusion matrix like
```
TN | FP
FN | TP
```
like this is because in their code, they have considered 0 to be the negative class and one to be positive class. sklearn always considers the smaller number to be negative and large number to positive. By number, I mean the class value (0 or 1). The order depends on your dataset and class. The accuracy will be the sum of diagonal elements divided by the sum of all the elements.p The diagonal elements are the number of correct predictions.


Question: Random forest class_weight and sample_weight parameters I have a class imbalance problem and been experimenting with a weighted Random Forest using the implementation in scikit-learn (>= 0.16). I have noticed that the implementation takes a class_weight parameter in the tree constructor and sample_weight parameter in the fit method to help solve class imbalance. Those two seem to be multiplied though to decide a final weight. I have trouble understanding the following: In what stages of the tree construction/training/prediction are those weights used? I have seen some papers for weighted trees, but I am not sure what scikit implements. What exactly is the difference between class_weight and sample_weight?

Response: RandomForests are built on Trees, which are very well documented. Check how Trees use the sample weighting: User guide on decision trees - tells exactly what algorithm is used Decision tree API - explains how sample_weight is used by trees (which for random forests, as you have determined, is the product of class_weight and sample_weight). As for the difference between class_weight and sample_weight: much can be determined simply by the nature of their datatypes. sample_weight is 1D array of length n_samples, assigning an explicit weight to each example used for training. class_weight is either a dictionary of each class to a uniform weight for that class (e.g., {1:.9, 2:.5, 3:.01}), or is a string telling sklearn how to automatically determine this dictionary. So the training weight for a given example is the product of it's explicitly named sample_weight (or 1 if sample_weight is not provided), and it's class_weight (or 1 if class_weight is not provided).


Question:
scikit learn - feature importance calculation in decision trees I'm trying to understand how feature importance is calculated for decision trees in sci-kit learn. This question has been asked before, but I am unable to reproduce the results the algorithm is providing. For example:
```
from StringIO import StringIO

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
from sklearn.tree.export import export_graphviz
from sklearn.feature_selection import mutual_info_classif

X = [[1,0,0], [0,0,0], [0,0,1], [0,1,0]]

y = [1,0,1,1]

clf = DecisionTreeClassifier()
clf.fit(X, y)

feat_importance = clf.tree_.compute_feature_importances(normalize=False)
print("feat importance = " + str(feat_importance))

out = StringIO()
out = export_graphviz(clf, out_file='test/tree.dot')
```
results in feature importance:
```
feat importance = [0.25       0.08333333 0.04166667]
```
and gives the following decision tree: Now, this answer to a similar question suggests the importance is calculated as Where G is the node impurity, in this case the gini impurity. This is the impurity reduction as far as I understood it. However, for feature 1 this should be: This answer suggests the importance is weighted by the probability of reaching the node (which is approximated by the proportion of samples reaching that node). Again, for feature 1 this should be: Both formulas provide the wrong result. How is the feature importance calculated correctly?

Response: I think feature importance depends on the implementation so we need to look at the documentation of scikit-learn. The feature importances. The higher, the more important the feature. The importance of a feature is computed as the (normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance That reduction or weighted information gain is defined as : The weighted impurity decrease equation is the following: N_t / N * (impurity - N_t_R / N_t * right_impurity - N_t_L / N_t * left_impurity) where N is the total number of samples, N_t is the number of samples at the current node, N_t_L is the number of samples in the left child, and N_t_R is the number of samples in the right child. http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier Since each feature is used once in your case, feature information must be equal to equation above. For X[2] : feature_importance = (4 / 4) * (0.375 - (0.75 * 0.444)) = 0.042 For X[1] : feature_importance = (3 / 4) * (0.444 - (2/3 * 0.5)) = 0.083 For X[0] : feature_importance = (2 / 4) * (0.5) = 0.25


Question: Exhaustively feature selection in scikit-learn? Is there any built in way of doing brute-force feature selection in scikit-learn, i.e. exhaustively evaluate all possible combinations of the input features, and then find the best subset? I am familiar with the "Recursive feature elimination" class but I am specifically interesting in evaluating all possible combinations of the input features one after the other.

Response:
No, best subset selection is not implemented. The easiest way to do it is to write it yourself. This should get you started:
```
from itertools import chain, combinations
from sklearn.cross_validation import cross_val_score

def best_subset_cv(estimator, X, y, cv=3):
    n_features = X.shape[1]
    subsets = chain.from_iterable(combinations(xrange(k), k + 1)
                                  for k in xrange(n_features))

    best_score = -np.inf
    best_subset = None
    for subset in subsets:
        score = cross_val_score(estimator, X[:, subset], y, cv=cv).mean()
        if score > best_score:
            best_score, best_subset = score, subset

    return best_subset, best_score
```
This performs k-fold cross-validation inside the loop, so it will fit k 2 ᵖ estimators when giving data with p features.


Question:
Lasso on sklearn does not converge When I run something like
```
import numpy
from sklearn import linear_model
A= #something
b= #something
clf=linear_model.Lasso(alpha=0.015, fit_intercept=False, tol=0.00000000000001,
          max_iter=10000000000000, positive=True)
clf.fit(A,b)
```
I get the error:
```
usr/local/lib/python2.7/dist-packages/scikit_learn-0.14.1-py2.7-linux-x86_64.egg/
sklearn/linear_model/coordinate_descent.py:418: UserWarning: Objective did not
converge. You might want to increase the number of iterations
' to increase the number of iterations')
```
The interesting thing is that A is never rank defficient. (I think)

Response: Try increasing tol. From the documentation: tol : float, optional The tolerance for the optimization: if the updates are smaller than tol, the optimization code checks the dual gap for optimality and continues until it is smaller than tol. The default for tol is 0.0001 on my version of scikit-learn. I assume that your tolerance is so small that the optimization never reaches a lower value.


Question:
Sklearn - How to predict probability for all target labels I have a data set with a target variable that can have 7 different labels. Each sample in my training set has only one label for the target variable. For each sample, I want to calculate the probability for each of the target labels. So my prediction would consist of 7 probabilities for each row. On the sklearn website I read about multi-label classification, but this doesn't seem to be what I want. I tried the following code, but this only gives me one classification per sample.
```
from sklearn.multiclass import OneVsRestClassifier
clf = OneVsRestClassifier(DecisionTreeClassifier())
clf.fit(X_train, y_train)
pred = clf.predict(X_test)
```
Does anyone have some advice on this? Thanks!

Response:
You can do that by simply removing the OneVsRestClassifer and using predict_proba method of the DecisionTreeClassifier. You can do the following:
```
clf = DecisionTreeClassifier()
clf.fit(X_train, y_train)
pred = clf.predict_proba(X_test)
```
This will give you a probability for each of your 7 possible classes. Hope that helps!


Question: Feature importances - Bagging, scikit-learn For a project I am comparing a number of decision trees, using the regression algorithms (Random Forest, Extra Trees, Adaboost and Bagging) of scikit-learn. To compare and interpret them I use the feature importance , though for the bagging decision tree this does not look to be available. My question: Does anybody know how to get the feature importances list for Bagging? Greetings, Kornee

Response:
Are you talking about BaggingClassifier? It can be used with many base estimators, so there is no feature importances implemented. There are model-independent methods for computing feature importances (see e.g. https://github.com/scikit-learn/scikit-learn/issues/8898), scikit-learn doesn't use them. In case of decision trees as base estimators you can compute feature importances yourselves: it'd be just an average of tree.feature_importances_ among all trees in bagging.estimators_:
```
import numpy as np
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
clf = BaggingClassifier(DecisionTreeClassifier())
clf.fit(X, y)

feature_importances = np.mean([
    tree.feature_importances_ for tree in clf.estimators_
], axis=0)
```
RandomForestClassifer does the same computation internally.


Question:
Feature names from OneHotEncoder I am using OneHotEncoder to encode few categorical variables (eg - Sex and AgeGroup). The resulting feature names from the encoder are like - 'x0_female', 'x0_male', 'x1_0.0', 'x1_15.0' etc.
```

```python
train_X = pd.DataFrame({'Sex':['male', 'female']*3, 'AgeGroup':[0,15,30,45,60,75]})
```

```python
from sklearn.preprocessing import OneHotEncoder
```

```python
encoder = OneHotEncoder()
```

```python
train_X_encoded = encoder.fit_transform(train_X[['Sex', 'AgeGroup']])
#Output
#```
#```
```

```python
encoder.get_feature_names()
```

```python
array(['x0_female', 'x0_male', 'x1_0.0', 'x1_15.0', 'x1_30.0', 'x1_45.0',
#Output
#       'x1_60.0', 'x1_75.0'], dtype=object)
#```
#Is there a way to tell OneHotEncoder to create the feature names in such a way that the column name is added at the beginning, something like - Sex_female, AgeGroup_15.0 etc, similar to what Pandas get_dummies() does.
```

Response:
A list with the original column names can be passed to get_feature_names.
```py

```python
encoder.get_feature_names(['Sex', 'AgeGroup'])
```

array(['Sex_female', 'Sex_male', 'AgeGroup_0', 'AgeGroup_15',
       'AgeGroup_30', 'AgeGroup_45', 'AgeGroup_60', 'AgeGroup_75'],
      dtype=object)
```
DEPRECATED: get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead. As per sklearn.preprocessing.OneHotEncoder.
```py

```python
encoder.get_feature_names_out(['Sex', 'AgeGroup'])
```

array(['Sex_female', 'Sex_male', 'AgeGroup_0', 'AgeGroup_15',
       'AgeGroup_30', 'AgeGroup_45', 'AgeGroup_60', 'AgeGroup_75'],
      dtype=object)
```


Question:
Trouble fitting simple data with MLPRegressor I am trying out Python and scikit-learn. I cannot get MLPRegressor to come even close to the data. Where is this going wrong?
```
from sklearn.neural_network import MLPRegressor
import numpy as np
import matplotlib.pyplot as plt

x = np.arange(0.0, 1, 0.01).reshape(-1, 1)
y = np.sin(2 * np.pi * x).ravel()

reg = MLPRegressor(hidden_layer_sizes=(10,),  activation='relu', solver='adam',    alpha=0.001,batch_size='auto',
               learning_rate='constant', learning_rate_init=0.01, power_t=0.5, max_iter=1000, shuffle=True,
               random_state=None, tol=0.0001, verbose=False, warm_start=False, momentum=0.9,
               nesterovs_momentum=True, early_stopping=False, validation_fraction=0.1, beta_1=0.9, beta_2=0.999,
               epsilon=1e-08)

reg = reg.fit(x, y)

test_x = np.arange(0.0, 1, 0.05).reshape(-1, 1)
test_y = reg.predict(test_x)

fig = plt.figure()
ax1 = fig.add_subplot(111)
ax1.scatter(x, y, s=10, c='b', marker="s", label='real')
ax1.scatter(test_x,test_y, s=10, c='r', marker="o", label='NN Prediction')

plt.show()
```
The result is not very good: Thank you.

Response:
You just need to change the solver to 'lbfgs'. The default'adam' is a SGD-like method, which is effective for large & messy data but pretty useless for this kind of smooth & small data. use a smooth activation function such as tanh. relu is almost linear, not suited for learning this simple non-linear function. Here're the result and the complete code. Even just 3 hidden neurons can achieve very high accuracy.
```
from sklearn.neural_network import MLPRegressor
import numpy as np
import matplotlib.pyplot as plt

x = np.arange(0.0, 1, 0.01).reshape(-1, 1)
y = np.sin(2 * np.pi * x).ravel()

nn = MLPRegressor(hidden_layer_sizes=(3),
                  activation='tanh', solver='lbfgs')

n = nn.fit(x, y)
test_x = np.arange(-0.1, 1.1, 0.01).reshape(-1, 1)
test_y = nn.predict(test_x)
fig = plt.figure()
ax1 = fig.add_subplot(111)
ax1.scatter(x, y, s=5, c='b', marker="o", label='real')
ax1.plot(test_x,test_y, c='r', label='NN Prediction')

plt.legend()
plt.show()
```


Question: One-Hot-Encode categorical variables and scale continuous ones simultaneouely I'm confused because it's going to be a problem if you first do OneHotEncoder and then StandardScaler because the scaler will also scale the columns previously transformed by OneHotEncoder. Is there a way to perform encoding and scaling at the same time and then concatenate the results together?

Response:
Sure thing. Just separately scale and one-hot-encode the separate columns as needed:
```
# Import libraries and download example data
from sklearn.preprocessing import StandardScaler, OneHotEncoder

dataset = pd.read_csv("https://stats.idre.ucla.edu/stat/data/binary.csv")
print(dataset.head(5))

# Define which columns should be encoded vs scaled
columns_to_encode = ['rank']
columns_to_scale  = ['gre', 'gpa']

# Instantiate encoder/scaler
scaler = StandardScaler()
ohe    = OneHotEncoder(sparse=False)

# Scale and Encode Separate Columns
scaled_columns  = scaler.fit_transform(dataset[columns_to_scale])
encoded_columns =    ohe.fit_transform(dataset[columns_to_encode])

# Concatenate (Column-Bind) Processed Columns Back Together
processed_data = np.concatenate([scaled_columns, encoded_columns], axis=1)
```


Question:
Scikit-learn Ridge classifier: extracting class probabilities I'm currently using sklearn's Ridge classifier, and am looking to ensemble this classifier with classifiers from sklearn and other libraries. In order to do this, it would be ideal to extract the probability that a given input belongs to each class in a list of classes. Currently, I'm zipping the classes with the output of model.decision_function(x), but this returns the distance from the hyperplane as opposed to a straightforward probability. These distance values vary from around -1 to around 1.
```
distances = dict(zip(clf.classes_, clf.decision_function(x)[0]))
```
How can I convert these distances to a more concrete set of probabilities (a series of positive values that sum to 1)? I'm looking for something like clf.predict_proba() that is implemented for the SVC in sklearn.

Response:
A little look at the source code of predict shows that decision_function is in fact the logit-transform of the actual class probabilities, i.e., if decision funciton is f, then the class probability of class 1 is exp(f) / (1 + exp(f)). This translates to following check in the sklearn source:
```
scores = self.decision_function(X)
    if len(scores.shape) == 1:
        indices = (scores > 0).astype(np.int)
    else:
        indices = scores.argmax(axis=1)
    return self.classes_[indices]
```
If you observe this check, it tells you that if decision function is greater than zero, then predict class 1, otherwise predict class 0 - a classical logit approach. So, you will have to turn the decision function into something like:
```
d = clf.decision_function(x)[0]
probs = numpy.exp(d) / (1 + numpy.exp(d))
```
And then take appropriate zip etc.


Question:
Sklearn changing string class label to int I have a pandas dataframe and I'm trying to change the values in a given column which are represented by strings into integers. For instance:
```
df = index    fruit   quantity   price
         0    apple          5    0.99
         1    apple          2    0.99
         2   orange          4    0.89
         4   banana          1    1.64
       ...
     10023     kiwi         10    0.92
```
I would like it to look at:
```
df = index    fruit   quantity   price
         0        1          5    0.99
         1        1          2    0.99
         2        2          4    0.89
         4        3          1    1.64
       ...
     10023        5         10    0.92
```
I can do this using
```
df["fruit"] = df["fruit"].map({"apple": 1, "orange": 2,...})
```
which works if I have a small list to change, but I'm looking at a column with over 500 different labels. Is there any way of changing this from a string to a an int?

Response:
You can use sklearn.preprocessing
```
from sklearn import preprocessing

le = preprocessing.LabelEncoder()
le.fit(df.fruit)
df['categorical_label'] = le.transform(df.fruit)
```
Transform labels back to original encoding.
```
le.inverse_transform(df['categorical_label'])
```


Question: scikit learn: train_test_split, can I ensure same splits on different datasets I understand that the train_test_split method splits a dataset into random train and test subsets. And using random_state=int can ensure we have the same splits on this dataset for each time the method is called. My problem is slightly different. I have two datasets, A and B, they contain identical sets of examples and the order of these examples appear in each dataset is also identical. But they key difference is that exmaples in each dataset uses a different sets of features. I would like to test to see if the features used in A leads to better performance than features used in B. So I would like to ensure that when I call train_test_split on A and B, I can get the same splits on both datasets so that the comparison is meaningful. Is this possible? Do I simply need to ensure the random_state in both method calls for both datasets are the same? Thanks

Response:
Yes, random state is enough.
```

```python
X, y = np.arange(10).reshape((5, 2)), range(5)
```

```python
X2 = np.hstack((X,X))
```

```python
X_train, X_test, _, _ = train_test_split(X,y, test_size=0.33, random_state=42)
```

```python
X_train2, X_test2, _, _ = train_test_split(X2,y, test_size=0.33, random_state=42)
```

```python
X_train
#Output
#array([[4, 5],
#       [0, 1],
#       [6, 7]])
```

```python
X_train2
#Output
#array([[4, 5, 4, 5],
#       [0, 1, 0, 1],
#       [6, 7, 6, 7]])
```

```python
X_test
#Output
#array([[2, 3],
#       [8, 9]])
```

```python
X_test2
#Output
#array([[2, 3, 2, 3],
#       [8, 9, 8, 9]])
#```
```


Question: Get nearest point to centroid, scikit-learn? I am using K-means for a clustering problem. I am trying to find the data point which is most close to the centroid, which I believe is called the medoid. Is there a way to do this in scikit-learn?

Response:
This is not the medoid, but here's something you can try:
```

```python
import numpy as np
```

```python
from sklearn.cluster import KMeans
```

```python
from sklearn.metrics import pairwise_distances_argmin_min
```

```python
X = np.random.randn(10, 4)
```

```python
km = KMeans(n_clusters=2).fit(X)
```

```python
closest, _ = pairwise_distances_argmin_min(km.cluster_centers_, X)
```

```python
closest
#Output
#array([0, 8])
#```
#The array closest contains the index of the point in X that is closest to each centroid. So X[0] is the closest point in X to centroid 0, and X[8] is the closest to centroid 1.
```


Question: Missing values in scikits machine learning Is it possible to have missing values in scikit-learn ? How should they be represented? I couldn't find any documentation about that.

Response: Missing values are simply not supported in scikit-learn. There has been discussion on the mailing list about this before, but no attempt to actually write code to handle them. Whatever you do, don't use NaN to encode missing values, since many of the algorithms refuse to handle samples containing NaNs. The above answer is outdated; the latest release of scikit-learn has a class Imputer that does simple, per-feature missing value imputation. You can feed it arrays containing NaNs to have those replaced by the mean, median or mode of the corresponding feature.


Question:
Factor Loadings using sklearn I want the correlations between individual variables and principal components in python. I am using PCA in sklearn. I don't understand how can I achieve the loading matrix after I have decomposed my data? My code is here.
```
iris = load_iris()
data, y = iris.data, iris.target
pca = PCA(n_components=2)
transformed_data = pca.fit(data).transform(data)
eigenValues = pca.explained_variance_ratio_
```
http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html doesn't mention how this can be achieved.

Response: I think that @RickardSjogren is describing the eigenvectors, while @BigPanda is giving the loadings. There's a big difference: Loadings vs eigenvectors in PCA: when to use one or another?. I created this PCA class with a loadings method. Loadings, as given by pca.components_ * np.sqrt(pca.explained_variance_), are more analogous to coefficients in a multiple linear regression. I don't use .T here because in the PCA class linked above, the components are already transposed. numpy.linalg.svd produces u, s, and vt, where vt is the Hermetian transpose, so you first need to back into v with vt.T. There is also one other important detail: the signs (positive/negative) on the components and loadings in sklearn.PCA may differ from packages such as R. More on that here: In sklearn.decomposition.PCA, why are components_ negative?.


Question: Can sklearn random forest directly handle categorical features? Say I have a categorical feature, color, which takes the values ['red', 'blue', 'green', 'orange'], and I want to use it to predict something in a random forest. If I one-hot encode it (i.e. I change it to four dummy variables), how do I tell sklearn that the four dummy variables are really one variable? Specifically, when sklearn is randomly selecting features to use at different nodes, it should either include the red, blue, green and orange dummies together, or it shouldn't include any of them. I've heard that there's no way to do this, but I'd imagine there must be a way to deal with categorical variables without arbitrarily coding them as numbers or something like that.

Response: No, there isn't. Somebody's working on this and the patch might be merged into mainline some day, but right now there's no support for categorical variables in scikit-learn except dummy (one-hot) encoding.


Question:
scikit-learn: how to scale back the 'y' predicted result I'm trying to learn scikit-learn and Machine Learning by using the Boston Housing Data Set.
```
# I splitted the initial dataset ('housing_X' and 'housing_y')
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(housing_X, housing_y, test_size=0.25, random_state=33)

# I scaled those two datasets
from sklearn.preprocessing import StandardScaler
scalerX = StandardScaler().fit(X_train)
scalery = StandardScaler().fit(y_train)
X_train = scalerX.transform(X_train)
y_train = scalery.transform(y_train)
X_test = scalerX.transform(X_test)
y_test = scalery.transform(y_test)

# I created the model
from sklearn import linear_model
clf_sgd = linear_model.SGDRegressor(loss='squared_loss', penalty=None, random_state=42)
train_and_evaluate(clf_sgd,X_train,y_train)
```
Based on this new model clf_sgd, I am trying to predict the y based on the first instance of X_train.
```
X_new_scaled = X_train[0]
print (X_new_scaled)
y_new = clf_sgd.predict(X_new_scaled)
print (y_new)
```
However, the result is quite odd for me (1.34032174, instead of 20-30, the range of the price of the houses)
```
[-0.32076092  0.35553428 -1.00966618 -0.28784917  0.87716097  1.28834383
  0.4759489  -0.83034371 -0.47659648 -0.81061061 -2.49222645  0.35062335
 -0.39859013]
[ 1.34032174]
```
I guess that this 1.34032174 value should be scaled back, but I am trying to figure out how to do it with no success. Any tip is welcome. Thank you very much.

Response:
You can use inverse_transform using your scalery object:
```
y_new_inverse = scalery.inverse_transform(y_new)
```


Question: Finding coefficients for logistic regression I'm working on a classification problem and need the coefficients of the logistic regression equation. I can find the coefficients in R but I need to submit the project in python. How to get the coefficient values in scikit-learn?

Response:
The statsmodels library would give you a breakdown of the coefficient results, as well as the associated p-values to determine their significance. Using an example of x1 and y1 variables:
```
x1_train, x1_test, y1_train, y1_test = train_test_split(x1, y1, random_state=0)

logreg = LogisticRegression().fit(x1_train,y1_train)
logreg

print("Training set score: {:.3f}".format(logreg.score(x1_train,y1_train)))
print("Test set score: {:.3f}".format(logreg.score(x1_test,y1_test)))

import statsmodels.api as sm
logit_model=sm.Logit(y1,x1)
result=logit_model.fit()
print(result.summary())
```
Example results:
```
Optimization terminated successfully.
         Current function value: 0.596755
         Iterations 7
                           Logit Regression Results
==============================================================================
Dep. Variable:             IsCanceled   No. Observations:                20000
Model:                          Logit   Df Residuals:                    19996
Method:                           MLE   Df Model:                            3
Date:                Sat, 17 Aug 2019   Pseudo R-squ.:                  0.1391
Time:                        23:58:55   Log-Likelihood:                -11935.
converged:                       True   LL-Null:                       -13863.
                                        LLR p-value:                     0.000
==============================================================================
                 coef    std err          z      P>|z|      [0.025      0.975]
------------------------------------------------------------------------------
const         -2.1417      0.050    -43.216      0.000      -2.239      -2.045
x1             0.0055      0.000     32.013      0.000       0.005       0.006
x2             0.0236      0.001     36.465      0.000       0.022       0.025
x3             2.1137      0.104     20.400      0.000       1.911       2.317
==============================================================================
```


Question: What does the CV stand for in sklearn.linear_model.LogisticRegressionCV? scikit-learn has two logistic regression functions: sklearn.linear_model.LogisticRegression sklearn.linear_model.LogisticRegressionCV I'm just curious what the CV stands for in the second one. The only acronym I know in ML that matches "CV" is cross-validation, but I'm guessing that's not it, since that would be achieved in scikit-learn with a wrapper function, not as part of the logistic regression function itself (I think).

Response: You are right in guessing that the latter allows the user to perform cross validation. The user can pass the number of folds as an argument cv of the function to perform k-fold cross-validation (default is 10 folds with StratifiedKFold). I would recommend reading the documentation for the functions LogisticRegression and LogisticRegressionCV


Question:
[sklearn][standardscaler] can I inverse the standardscaler for the model output? I have some data structured as below, trying to predict t from the features.
```
train_df

t: time to predict
f1: feature1
f2: feature2
f3:......
```
Can t be scaled with StandardScaler, so I instead predict t' and then inverse the StandardScaler to get back the real time? For example:
```
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(train_df['t'])
train_df['t']= scaler.transform(train_df['t'])
```
After this, I would like to: run regression model check the score check predicted t' with the real time value by using the inverse StandardScaler Is this possible?

Response: Yeah, and it's conveniently called inverse_transform. The documentation provides examples of its use.


Question: how to explain the decision tree from scikit-learn I have two problems with understanding the result of decision tree from scikit-learn. For example, this is one of my decision trees: My question is that how I can use the tree? The first question is that: if a sample satisfied the condition, then it goes to the LEFT branch (if exists), otherwise it goes RIGHT. In my case, if a sample with X[7] > 63521.3984. Then the sample will go to the green box. Correct? The second question is that: when a sample reaches the leaf node, how can I know which category it belongs? In this example, I have three categories to classify. In the red box, there are 91, 212, and 113 samples are satisfied the condition, respectively. But how can I decide the category? I know there is a function clf.predict(sample) to tell the category. Can I do that from the graph??? Many thanks.

Response: The value line in each box is telling you how many samples at that node fall into each category, in order. That's why, in each box, the numbers in value add up to the number shown in sample. For instance, in your red box, 91+212+113=416. So this means if you reach this node, there were 91 data points in category 1, 212 in category 2, and 113 in category 3. If you were going to predict the outcome for a new data point that reached that leaf in the decision tree, you would predict category 2, because that is the most common category for samples at that node.


Question: How to give column names after one-hot encoding with sklearn? Here is my question, I hope someone can help me to figure it out.. To explain, there are more than 10 categorical columns in my data set and each of them has 200-300 categories. I want to convert them into binary values. For that I used first label encoder to convert string categories into numbers. The Label Encoder code and the output is shown below. After Label Encoder, I used One Hot Encoder From scikit-learn again and it is worked. BUT THE PROBLEM IS, I need column names after one hot encoder. For example, column A with categorical values before encoding. A = [1,2,3,4,..] It should be like that after encoding, A-1, A-2, A-3 Anyone know how to assign column names to (old column names -value name or number) after one hot encoding. Here is my one hot encoding and it's output; I need columns with name because I trained an ANN, but every time data comes up I cannot convert all past data again and again. So, I want to add just new ones every time. Thank anyway..

Response:
You can get the column names using .get_feature_names() attribute.
```

```python
ohenc.get_feature_names()
```

```python
x_cat_df.columns = ohenc.get_feature_names()
#Output
#```
#Detailed example is here. Update from Version 1.0, use get_feature_names_out
```


Question:
efficiently passing dataframes as y and X to scikit-learn fits I generate a pandas dataframe from read_sql_query. It has three columns, "results, speed, weight" I want to use scikit-learn LinearRegression to fit results = f(speed, weight) I haven't been able to find the correct syntax that would allow me to pass this dataframe, or column slices of it, to LinearRegression.fit(y, X).
```
print df['result'].shape
print df[['speed', 'weight']].shape
(8L,)
(8, 2)
```
but I cannot pass that to fit
```
lm.fit(df['result'], df[['speed', 'weight']])
```
It throws a deprecation warning and a ValueError
```
DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and willraise ValueError in 0.19.
ValueError: Found arrays with inconsistent numbers of samples: [1 8]
```
What is the efficient, clean way to take dataframes of targets and features, and pass them to fit operations? This is how I generated the example:
```
import pandas as pd
import numpy as np
from datetime import datetime, timedelta

date_today = datetime.now()
days = pd.date_range(date_today, date_today + timedelta(7), freq='D')

np.random.seed(seed=1111)
data = np.random.randint(1, high=100, size=len(days))
data2 = np.random.randint(1, high=100, size=len(days))
data3 = np.random.randint(1, high=100, size=len(days))
df = pd.DataFrame({'test': days, 'result': data,'speed': data2,'weight': data3})
df = df.set_index('test')
print(df)
```

Response:
You are sending values in incorrect order. All scikit-learn estimators implementing fit() accept input X, y not y, X as you are doing. Try this:
```
lm.fit(df[['speed', 'weight']], df['result'])
```


Question:
Removing features with low variance using scikit-learn scikit-learn provides various methods to remove descriptors, a basic method for this purpose has been provided by the given tutorial below, http://scikit-learn.org/stable/modules/feature_selection.html but the tutorial does not provide any method or a way that can tell you the way to keep the list of features that either removed or kept. The code below has been taken from the tutorial.
```
from sklearn.feature_selection import VarianceThreshold
    X = [[0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 1, 1], [0, 1, 0], [0, 1, 1]]
    sel = VarianceThreshold(threshold=(.8 * (1 - .8)))
    sel.fit_transform(X)
array([[0, 1],
       [1, 0],
       [0, 0],
       [1, 1],
       [1, 0],
       [1, 1]])
```
The given example code above depicts only two descriptors "shape(6, 2)", but in my case, I have a huge data frames with a shape of (rows 51, columns 9000). After finding a suitable model I want to keep the track of useful and useless features because I can save computational time during the computation of the features of test data set by calculating only useful features. For example, when you perform machine learning modeling with WEKA 6.0, it provided with remarkable flexibility over feature selection and after removing the useless feature you can get a list of a discarded features along with the useful features. thanks

Response:
Then, what you can do, if I'm not wrong is: In the case of the VarianceThreshold, you can call the method fit instead of fit_transform. This will fit data, and the resulting variances will be stored in vt.variances_ (assuming vt is your object). Having a threhold, you can extract the features of the transformation as fit_transform would do:
```
X[:, vt.variances_ > threshold]
```
Or get the indexes as:
```
idx = np.where(vt.variances_ > threshold)[0]
```
Or as a mask
```
mask = vt.variances_ > threshold
```
PS: default threshold is 0 EDIT: A more straight forward to do, is by using the method get_support of the class VarianceThreshold. From the documentation:
```
get_support([indices])  Get a mask, or integer index, of the features selected
```
You should call this method after fit or fit_transform.


Question:
Get feature importance from GridSearchCV Is there a way to get feature importance from a sklearn's GridSearchCV? For example :
```
from sklearn.model_selection import GridSearchCV
print("starting grid search ......")
optimized_GBM = GridSearchCV(LGBMRegressor(),
                             params,
                             cv=3,
                             n_jobs=-1)
#
optimized_GBM.fit(tr, yvar)
preds2 = optimized_GBM.predict(te)
```
Is there a way I can access feature importance ? Maybe something like
```
optimized_GBM.feature_importances_
```

Response:
This one works
```
optimized_GBM.best_estimator_.feature_importances_
```


Question:
Keep pandas structure with numpy/scikit functions I'm using the excellent read_csv()function from pandas, which gives:
```

```python
data = pandas.read_csv("lala.csv", delimiter=",")
```

```python
data
#Output
#<class 'pandas.core.frame.DataFrame'>
#Int64Index: 12083 entries, 0 to 12082
#Columns: 569 entries, REGIONC to SCALEKER
#dtypes: float64(51), int64(518)
#```
#but when i apply a function from scikit-learn i loose the informations about columns:
#```
#from sklearn import preprocessing
#preprocessing.scale(data)
#```
#gives numpy array. Is there a way to apply scikit or numpy function to DataFrames without loosing the information?
```

Response:
This can be done by wrapping the returned data in a dataframe, with index and columns information in.
```
import pandas as pd
pd.DataFrame(preprocessing.scale(data), index = data.index, columns = data.columns)
```


Question:
find the "elbow point" on an optimization curve with Python i have a list of points which are the inertia values of a kmeans algorithm. To determine the optimum amount of clusters i need to find the point, where this curve starts to flatten. Data example Here is how my list of values is created and filled:
```
sum_squared_dist = []
K = range(1,50)
for k in K:
    km = KMeans(n_clusters=k, random_state=0)
    km = km.fit(normalized_modeling_data)
    sum_squared_dist.append(km.inertia_)

print(sum_squared_dist)
```
How can i find a point, where the pitch of this curve increases (the curve is falling, so the first derivation is negative)? My approach
```
derivates = []
for i in range(len(sum_squared_dist)):
    derivates.append(sum_squared_dist[i] - sum_squared_dist[i-1])
```
I want to find the optimum number of clusters any given data using the elbow method. Could someone help me how i can find the point where the list of the inertia values starts to flatten? Edit Datapoints:
```
[7342.1301373073857, 6881.7109460930769, 6531.1657905495022,
6356.2255554679778, 6209.8382535595829, 6094.9052166741121,
5980.0191582610196, 5880.1869867848218, 5779.8957906367368,
5691.1879324562778, 5617.5153566271356, 5532.2613232619951,
5467.352265375117, 5395.4493783888756, 5345.3459908298091,
5290.6769823693812, 5243.5271656371888, 5207.2501206569532,
5164.9617535255456]
```
Graph:

Response:
I worked on a Python package modeled after the Kneedle algorithm. It finds x=5 as the point where the curve starts to flatten. The documentation and the paper discuss the algorithm for choosing the knee point in more detail.
```
y = [7342.1301373073857, 6881.7109460930769, 6531.1657905495022,
6356.2255554679778, 6209.8382535595829, 6094.9052166741121,
5980.0191582610196, 5880.1869867848218, 5779.8957906367368,
5691.1879324562778, 5617.5153566271356, 5532.2613232619951,
5467.352265375117, 5395.4493783888756, 5345.3459908298091,
5290.6769823693812, 5243.5271656371888, 5207.2501206569532,
5164.9617535255456]

x = range(1, len(y)+1)

from kneed import KneeLocator
kn = KneeLocator(x, y, curve='convex', direction='decreasing')
print(kn.knee)
5

import matplotlib.pyplot as plt
plt.xlabel('number of clusters k')
plt.ylabel('Sum of squared distances')
plt.plot(x, y, 'bx-')
plt.vlines(kn.knee, plt.ylim()[0], plt.ylim()[1], linestyles='dashed')
```


Question:
Facing ValueError: Target is multiclass but average='binary' I'm trying to use Naive Bayes algorithm for my dataset. I'm able to find out the accuracy but trying to find out precision and recall for the same. But, it is throwing the following error:
```none
ValueError: Target is multiclass but average='binary'. Please choose another average setting.
```
Can anyone please suggest me how to proceed with it. I have tried using average ='micro' in the precision and the recall scores. It worked without any errors but it is giving the same score for accuracy, precision, recall. My dataset: train_data.csv:
```none
review,label
Colors & clarity is superb,positive
Sadly the picture is not nearly as clear or bright as my 40 inch Samsung,negative
```
test_data.csv:
```none
review,label
The picture is clear and beautiful,positive
Picture is not clear,negative
```
My code:
```py
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score
from sklearn.metrics import confusion_matrix

X_train, y_train = pd.read_csv('train_data.csv')
X_test, y_test = pd.read_csv('test_data.csv')

vec = CountVectorizer()
X_train_transformed = vec.fit_transform(X_train)
X_test_transformed = vec.transform(X_test)

clf = MultinomialNB()
clf.fit(X_train_transformed, y_train)

score = clf.score(X_test_transformed, y_test)

y_pred = clf.predict(X_test_transformed)
cm = confusion_matrix(y_test, y_pred)

precision = precision_score(y_test, y_pred, pos_label='positive')
recall = recall_score(y_test, y_pred, pos_label='positive')
```

Response:
You need to add the 'average' param. According to the documentation: average : string, [None, ‘binary’ (default), ‘micro’, ‘macro’, ‘samples’, ‘weighted’] This parameter is required for multiclass/multilabel targets. If None, the scores for each class are returned. Otherwise, this determines the type of averaging performed on the data: Do this:
```
print("Precision Score : ",precision_score(y_test, y_pred,
                                           pos_label='positive'
                                           average='micro'))
print("Recall Score : ",recall_score(y_test, y_pred,
                                           pos_label='positive'
                                           average='micro'))
```
Replace 'micro' with any one of the above options except 'binary'. Also, in the multiclass setting, there is no need to provide the 'pos_label' as it will be anyways ignored. Update for comment: Yes, they can be equal. Its given in the user guide here: Note that for “micro”-averaging in a multiclass setting with all labels included will produce equal precision, recall and F, while “weighted” averaging may produce an F-score that is not between precision and recall.


Question:
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample While I am predicting the one sample from my data, it gives reshape error but my model has equal number of rows. Here is my code:
```py
import pandas as pd
from sklearn.linear_model import LinearRegression
import numpy as np
x = np.array([2.0 , 2.4, 1.5, 3.5, 3.5, 3.5, 3.5, 3.7, 3.7])
y = np.array([196, 221, 136, 255, 244, 230, 232, 255, 267])

lr = LinearRegression()
lr.fit(x,y)

print(lr.predict(2.4))
```
The error is
```none
if it contains a single sample.".format(array))
ValueError: Expected 2D array, got scalar array instead:
array=2.4.
Reshape your data either using array.reshape(-1, 1) if your data has a
single feature or array.reshape(1, -1) if it contains a single sample.
```

Response:
You should reshape your X to be a 2D array not 1D array. Fitting a model requires requires a 2D array. i.e (n_samples, n_features)
```
x = np.array([2.0 , 2.4, 1.5, 3.5, 3.5, 3.5, 3.5, 3.7, 3.7])
y = np.array([196, 221, 136, 255, 244, 230, 232, 255, 267])

lr = LinearRegression()
lr.fit(x.reshape(-1, 1), y)

print(lr.predict([[2.4]]))
```


Question: scikit-learn .predict() default threshold I'm working on a classification problem with unbalanced classes (5% 1's). I want to predict the class, not the probability. In a binary classification problem, is scikit's classifier.predict() using 0.5 by default? If it doesn't, what's the default method? If it does, how do I change it? In scikit some classifiers have the class_weight='auto' option, but not all do. With class_weight='auto', would .predict() use the actual population proportion as a threshold? What would be the way to do this in a classifier like MultinomialNB that doesn't support class_weight? Other than using predict_proba() and then calculation the classes myself.

Response:
The threshold can be set using clf.predict_proba() for example:
```
from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier(random_state = 2)
clf.fit(X_train,y_train)
# y_pred = clf.predict(X_test)  # default threshold is 0.5
y_pred = (clf.predict_proba(X_test)[:,1] >= 0.3).astype(bool) # set threshold as 0.3
```


Question:
How to get feature Importance in naive bayes? I have a dataset of reviews which has a class label of positive/negative. I am applying Naive Bayes to that reviews dataset. Firstly, I am converting into Bag of words. Here sorted_data['Text'] is reviews and final_counts is a sparse matrix
```
count_vect = CountVectorizer()
final_counts = count_vect.fit_transform(sorted_data['Text'].values)
```
I am splitting the data into train and test dataset.
```
X_1, X_test, y_1, y_test = cross_validation.train_test_split(final_counts, labels, test_size=0.3, random_state=0)
```
I am applying the naive bayes algorithm as follows
```
optimal_alpha = 1
NB_optimal = BernoulliNB(alpha=optimal_aplha)

# fitting the model
NB_optimal.fit(X_tr, y_tr)

# predict the response
pred = NB_optimal.predict(X_test)

# evaluate accuracy
acc = accuracy_score(y_test, pred) * 100
print('\nThe accuracy of the NB classifier for k = %d is %f%%' % (optimal_aplha, acc))
```
Here X_test is test dataset in which pred variable gives us whether the vector in X_test is positive or negative class. The X_test shape is (54626 rows, 82343 dimensions) length of pred is 54626 My question is I want to get the words with highest probability in each vector so that I can get to know by the words that why it predicted as positive or negative class. Therefore, how to get the words which have highest probability in each vector?

Response:
You can get the importantance of each word out of the fit model by using the coefs_ or feature_log_prob_ attributes. For example
```
neg_class_prob_sorted = NB_optimal.feature_log_prob_[0, :].argsort()[::-1]
pos_class_prob_sorted = NB_optimal.feature_log_prob_[1, :].argsort()[::-1]

print(np.take(count_vect.get_feature_names(), neg_class_prob_sorted[:10]))
print(np.take(count_vect.get_feature_names(), pos_class_prob_sorted[:10]))
```
Prints the top 10 most predictive words for each of your classes.


Question:
When scale the data, why the train dataset use 'fit' and 'transform', but the test dataset only use 'transform'? When scale the data, why the train dataset use 'fit' and 'transform', but the test dataset only use 'transform'?
```
SAMPLE_COUNT = 5000
TEST_COUNT = 20000
seed(0)
sample = list()
test_sample = list()
for index, line in enumerate(open('covtype.data','rb')):
    if index < SAMPLE_COUNT:
        sample.append(line)
    else:
        r = randint(0,index)
        if r < SAMPLE_COUNT:
            sample[r] = line
        else:
            k = randint(0,index)
            if k < TEST_COUNT:
                if len(test_sample) < TEST_COUNT:
                    test_sample.append(line)
                else:
                    test_sample[k] = line
from sklearn.preprocessing import StandardScaler
for n, line in enumerate(sample):
sample[n] = map(float, line.strip().split(','))
y = np.array(sample)[:,-1]
scaling = StandardScaler()

X = scaling.fit_transform(np.array(sample)[:,:-1]) ##here use fit and transform

for n,line in enumerate(test_sample):
test_sample[n] = map(float,line.strip().split(','))
yt = np.array(test_sample)[:,-1]

Xt = scaling.transform(np.array(test_sample)[:,:-1])##why here only use transform
```
As the annotation says, why Xt only use transform but no fit?

Response: We use fit_transform() on the train data so that we learn the parameters of scaling on the train data and in the same time we scale the train data. We only use transform() on the test data because we use the scaling paramaters learned on the train data to scale the test data. This is the standart procedure to scale. You always learn your scaling parameters on the train and then use them on the test. Here is an article that explane it very well : https://sebastianraschka.com/faq/docs/scale-training-test.html


Question:
scikit-learn: clustering text documents using DBSCAN I'm tryin to use scikit-learn to cluster text documents. On the whole, I find my way around, but I have my problems with specific issues. Most of the examples I found illustrate clustering using scikit-learn with k-means as clustering algorithm. Adopting these example with k-means to my setting works in principle. However, k-means is not suitable since I don't know the number of clusters. From what I read so far -- please correct me here if needed -- DBSCAN or MeanShift seem the be more appropriate in my case. The scikit-learn website provides examples for each cluster algorithm. The problem is now, that with both DBSCAN and MeanShift I get errors I cannot comprehend, let alone solve. My minimal code is as follows:
```
docs = []
for item in [database]:
    docs.append(item)

vectorizer = TfidfVectorizer(min_df=1)
X = vectorizer.fit_transform(docs)

X = X.todense() # <-- This line was needed to resolve the isse

db = DBSCAN(eps=0.3, min_samples=10).fit(X)
...
```
(My documents are already processed, i.e., stopwords have been removed and an Porter Stemmer has been applied.) When I run this code, I get the following error when instatiating DBSCAN and calling fit():
```
...
File "/usr/local/lib/python2.7/dist-packages/sklearn/cluster/dbscan_.py", line 248, in fit
clust = dbscan(X, **self.get_params())
File "/usr/local/lib/python2.7/dist-packages/sklearn/cluster/dbscan_.py", line 86, in dbscan
n = X.shape[0]
IndexError: tuple index out of range
```
Clicking on the line in dbscan_.py that throws the error, I noticed the following line
```
...
X = np.asarray(X)
n = X.shape[0]
...
```
When I use these to lines directly in my code for testing, I get the same error. I don't really know what np.asarray(X) is doing here, but after the command X.shape = (). Hence X.shape[0] bombs -- before, X.shape[0] correctly refers to the number of documents. Out of curiosity, I removed X = np.asarray(X) from dbscan_.py. When I do this, something is computing heavily. But after some seconds, I get another error:
```
...
File "/usr/lib/python2.7/dist-packages/scipy/sparse/csr.py", line 214, in extractor
(min_indx,max_indx) = check_bounds(indices,N)
File "/usr/lib/python2.7/dist-packages/scipy/sparse/csr.py", line 198, in check_bounds
max_indx = indices.max()
File "/usr/lib/python2.7/dist-packages/numpy/core/_methods.py", line 17, in _amax
out=out, keepdims=keepdims)
ValueError: zero-size array to reduction operation maximum which has no identity
```
In short, I have no clue how to get DBSCAN working, or what I might have missed, in general.

Response: It looks like sparse representations for DBSCAN are supported as of Jan. 2015. I upgraded sklearn to 0.16.1 and it worked for me on text.


Question:
Python scikit learn MLPClassifier "hidden_layer_sizes" I am lost in the scikit learn 0.18 user manual (http://scikit-learn.org/dev/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier):
```
hidden_layer_sizes : tuple, length = n_layers - 2, default (100,)
   The ith element represents the number of neurons in the ith hidden layer.
```
If I am looking for only 1 hidden layer and 7 hidden units in my model, should I put like this? Thanks!
```
hidden_layer_sizes=(7, 1)
```

Response: hidden_layer_sizes=(7,) if you want only 1 hidden layer with 7 hidden units. length = n_layers - 2 is because you have 1 input layer and 1 output layer.


Question:
True Positive Rate and False Positive Rate (TPR, FPR) for Multi-Class Data in python [duplicate] This question already has answers here: How to get precision, recall and f-measure from confusion matrix in Python [duplicate] (3 answers) calculate precision and recall in a confusion matrix (5 answers) Closed 4 years ago. How do you compute the true- and false- positive rates of a multi-class classification problem? Say,
```
y_true = [1, -1,  0,  0,  1, -1,  1,  0, -1,  0,  1, -1,  1,  0,  0, -1,  0]
y_prediction = [-1, -1,  1,  0,  0,  0,  0, -1,  1, -1,  1,  1,  0,  0,  1,  1, -1]
```
The confusion matrix is computed by metrics.confusion_matrix(y_true, y_prediction), but that just shifts the problem. EDIT after @seralouk's answer. Here, the class -1 is to be considered as the negatives, while 0 and 1 are variations of positives.

Response:
Using your data, you can get all the metrics for all the classes at once:
```
import numpy as np
from sklearn.metrics import confusion_matrix

y_true = [1, -1,  0,  0,  1, -1,  1,  0, -1,  0,  1, -1,  1,  0,  0, -1,  0]
y_prediction = [-1, -1,  1,  0,  0,  0,  0, -1,  1, -1,  1,  1,  0,  0,  1,  1, -1]
cnf_matrix = confusion_matrix(y_true, y_prediction)
print(cnf_matrix)
#[[1 1 3]
# [3 2 2]
# [1 3 1]]

FP = cnf_matrix.sum(axis=0) - np.diag(cnf_matrix)
FN = cnf_matrix.sum(axis=1) - np.diag(cnf_matrix)
TP = np.diag(cnf_matrix)
TN = cnf_matrix.sum() - (FP + FN + TP)

FP = FP.astype(float)
FN = FN.astype(float)
TP = TP.astype(float)
TN = TN.astype(float)

# Sensitivity, hit rate, recall, or true positive rate
TPR = TP/(TP+FN)
# Specificity or true negative rate
TNR = TN/(TN+FP)
# Precision or positive predictive value
PPV = TP/(TP+FP)
# Negative predictive value
NPV = TN/(TN+FN)
# Fall out or false positive rate
FPR = FP/(FP+TN)
# False negative rate
FNR = FN/(TP+FN)
# False discovery rate
FDR = FP/(TP+FP)
# Overall accuracy
ACC = (TP+TN)/(TP+FP+FN+TN)
```
For a general case where we have a lot of classes, these metrics are represented graphically in the following image:


Question:
How to generate a train-test-split based on a group id? I have the following data:
```none
Group_ID Item_id Target
0         1       1      0
1         1       2      0
2         1       3      1
3         2       4      0
4         2       5      1
5         2       6      1
6         3       7      0
7         4       8      0
8         5       9      0
9         5      10      1
```
I need to split the dataset into a training and testing set based on the "Group_ID" so that 80% of the data goes into a training set and 20% into a test set. That is, I need my training set to look something like:
```none
Group_ID Item_id Target
0          1       1      0
1          1       2      0
2          1       3      1
3          2       4      0
4          2       5      1
5          2       6      1
6          3       7      0
7          4       8      0
```
And test set:
```none
Group_ID Item_id Target
8         5       9      0
9         5      10      1
```
What would be the simplest way to do this? As far as I know, the standard test_train_split function in sklearn does not support splitting by groups in a way where I can also indicate the size of the split (e.g. 80/20).

Response:
I figured out the answer. This seems to work:
```
from sklearn.model_selection import GroupShuffleSplit

splitter = GroupShuffleSplit(test_size=.20, n_splits=2, random_state = 7)
split = splitter.split(df, groups=df['Group_Id'])
train_inds, test_inds = next(split)

train = df.iloc[train_inds]
test = df.iloc[test_inds]
```


Question:
AttributeError: 'GridSearchCV' object has no attribute 'best_params_' Grid search is a way to find the best parameters for any model out of the combinations we specify. I have formed a grid search on my model in the below manner and wish to find best parameters identified using this gridsearch.
```py
from sklearn.model_selection import GridSearchCV
# Create the parameter grid based on the results of random search
param_grid = {
    'bootstrap': [True],'max_depth': [20,30,40, 100, 110],
    'max_features': ['sqrt'],'min_samples_leaf': [5,10,15],
    'min_samples_split': [40,50,60], 'n_estimators': [150, 200, 250]
}
# Create a based model
rf = RandomForestClassifier()
# Instantiate the grid search model
grid_search = GridSearchCV(estimator = rf, param_grid = param_grid,
                          cv = 3, n_jobs = -1, verbose = 2)
```
Now I want to find the best parameters of the gridsearch as the output
```py
grid_search.best_params_
```
Error:
```none
----> grid_search.best_params_
AttributeError: 'GridSearchCV' object has no attribute 'best_params_'
```
What am I missing?

Response:
You cannot get best parameters without fitting the data. Fit the data
```
grid_search.fit(X_train, y_train)
```
Now find the best parameters.
```
grid_search.best_params_
```
grid_search.best_params_ will work after fitting on X_train and y_train.


Question:
Extract blocks or patches from NumPy Array I have a 2-d numpy array as follows:
```
a = np.array([[1,5,9,13],
              [2,6,10,14],
              [3,7,11,15],
              [4,8,12,16]]
```
I want to extract it into patches of 2 by 2 sizes with out repeating the elements. The answer should exactly be the same. This can be 3-d array or list with the same order of elements as below:
```
[[[1,5],
 [2,6]],

 [[3,7],
 [4,8]],

 [[9,13],
 [10,14]],

 [[11,15],
 [12,16]]]
```
How can do it easily? In my real problem the size of a is (36, 72). I can not do it one by one. I want programmatic way of doing it.

Response:
Using scikit-image:
```
import numpy as np
from skimage.util import view_as_blocks

a = np.array([[1,5,9,13],
              [2,6,10,14],
              [3,7,11,15],
              [4,8,12,16]])

print(view_as_blocks(a, (2, 2)))
```


Question:
Grid-Search finding Parameters for AUC I'm trying to find the parameters for my SVM, which give me the best AUC. But i can't find any scoring function for AUC in sklearn. Does someone have an idea? Here is my Code:
```
parameters = {"C":[0.1, 1, 10, 100, 1000], "gamma":[0.1, 0.01, 0.001, 0.0001, 0.00001]}
    clf = SVC(kernel = "rbf")
    clf = GridSearchCV(clf, parameters, scoring = ???)
    svr.fit(features_train , labels_train)
    print svr.best_params_
```
So what can i use for ??? to get the best parameters for an high AUC score?

Response:
You can simply use:
```
clf = GridSearchCV(clf, parameters, scoring='roc_auc')
```


Question: Sklearn, gridsearch: how to print out progress during the execution? I am using GridSearch from sklearn to optimize parameters of the classifier. There is a lot of data, so the whole process of optimization takes a while: more than a day. I would like to watch the performance of the already-tried combinations of parameters during the execution. Is it possible?

Response:
Set the verbose parameter in GridSearchCV to a positive number (the greater the number the more detail you will get). For instance:
```
GridSearchCV(clf, param_grid, cv=cv, scoring='accuracy', verbose=10)
```


Question: Relationship between SciPy and NumPy SciPy appears to provide most (but not all [1]) of NumPy's functions in its own namespace. In other words, if there's a function named numpy.foo, there's almost certainly a scipy.foo. Most of the time, the two appear to be exactly the same, oftentimes even pointing to the same function object. Sometimes, they're different. To give an example that came up recently: numpy.log10 is a ufunc that returns NaNs for negative arguments; scipy.log10 returns complex values for negative arguments and doesn't appear to be a ufunc. The same can be said about log, log2 and logn, but not about log1p [2]. On the other hand, numpy.exp and scipy.exp appear to be different names for the same ufunc. This is also true of scipy.log1p and numpy.log1p. Another example is numpy.linalg.solve vs scipy.linalg.solve. They're similar, but the latter offers some additional features over the former. Why the apparent duplication? If this is meant to be a wholesale import of numpy into the scipy namespace, why the subtle differences in behaviour and the missing functions? Is there some overarching logic that would help clear up the confusion? [1] numpy.min, numpy.max, numpy.abs and a few others have no counterparts in the scipy namespace. [2] Tested using NumPy 1.5.1 and SciPy 0.9.0rc2.

Response:
Last time I checked it, the scipy __init__ method executes a
```
from numpy import *
```
so that the whole numpy namespace is included into scipy when the scipy module is imported. The log10 behavior you are describing is interesting, because both versions are coming from numpy. One is a ufunc, the other is a numpy.lib function. Why scipy is preferring the library function over the ufunc, I don't know off the top of my head. EDIT: In fact, I can answer the log10 question. Looking in the scipy __init__ method I see this:
```
# Import numpy symbols to scipy name space
import numpy as _num
from numpy import oldnumeric
from numpy import *
from numpy.random import rand, randn
from numpy.fft import fft, ifft
from numpy.lib.scimath import *
```
The log10 function you get in scipy comes from numpy.lib.scimath. Looking at that code, it says:
```
"""
Wrapper functions to more user-friendly calling of certain math functions
whose output data-type is different than the input data-type in certain
domains of the input.

For example, for functions like log() with branch cuts, the versions in this
module provide the mathematically valid answers in the complex plane:

```python
import math
```

```python
from numpy.lib import scimath
```

```python
scimath.log(-math.exp(1)) == (1+1j*math.pi)
#Output
#True
```

Similarly, sqrt(), other base logarithms, power() and trig functions are
correctly handled.  See their respective docstrings for specific examples.
"""
```
It seems that module overlays the base numpy ufuncs for sqrt, log, log2, logn, log10, power, arccos, arcsin, and arctanh. That explains the behavior you are seeing. The underlying design reason why it is done like that is probably buried in a mailing list post somewhere.


Question:
How to add a new row to an empty numpy array Using standard Python arrays, I can do the following:
```
arr = []
arr.append([1,2,3])
arr.append([4,5,6])
# arr is now [[1,2,3],[4,5,6]]
```
However, I cannot do the same thing in numpy. For example:
```
arr = np.array([])
arr = np.append(arr, np.array([1,2,3]))
arr = np.append(arr, np.array([4,5,6]))
# arr is now [1,2,3,4,5,6]
```
I also looked into vstack, but when I use vstack on an empty array, I get:
```
ValueError: all the input array dimensions except for the concatenation axis must match exactly
```
So how do I do append a new row to an empty array in numpy?

Response:
The way to "start" the array that you want is:
```
arr = np.empty((0,3), int)
```
Which is an empty array but it has the proper dimensionality.
```
>>> arr
array([], shape=(0, 3), dtype=int64)
```
Then be sure to append along axis 0:
```
arr = np.append(arr, np.array([[1,2,3]]), axis=0)
arr = np.append(arr, np.array([[4,5,6]]), axis=0)
```
But, @jonrsharpe is right. In fact, if you're going to be appending in a loop, it would be much faster to append to a list as in your first example, then convert to a numpy array at the end, since you're really not using numpy as intended during the loop:
```

```python
%%timeit
   .....: l = []
   .....: for i in xrange(1000):
   .....:     l.append([3*i+1,3*i+2,3*i+3])
   .....: l = np.asarray(l)
   .....:
1000 loops, best of 3: 1.18 ms per loop
```

```python
%%timeit
   .....: a = np.empty((0,3), int)
   .....: for i in xrange(1000):
   .....:     a = np.append(a, 3*i+np.array([[1,2,3]]), 0)
   .....:
100 loops, best of 3: 18.5 ms per loop
```

```python
np.allclose(a, l)
#Output
#True
#```
#The numpythonic way to do it depends on your application, but it would be more like:
#```
```

```python
timeit n = np.arange(1,3001).reshape(1000,3)
100000 loops, best of 3: 5.93 µs per loop
```

```python
np.allclose(a, n)
#Output
#True
#```
```


Question:
How to normalize a NumPy array to within a certain range? After doing some processing on an audio or image array, it needs to be normalized within a range before it can be written back to a file. This can be done like so:
```
# Normalize audio channels to between -1.0 and +1.0
audio[:,0] = audio[:,0]/abs(audio[:,0]).max()
audio[:,1] = audio[:,1]/abs(audio[:,1]).max()

# Normalize image to between 0 and 255
image = image/(image.max()/255.0)
```
Is there a less verbose, convenience function way to do this? matplotlib.colors.Normalize() doesn't seem to be related.

Response:
```
# Normalize audio channels to between -1.0 and +1.0
audio /= np.max(np.abs(audio),axis=0)
# Normalize image to between 0 and 255
image *= (255.0/image.max())
```
Using /= and *= allows you to eliminate an intermediate temporary array, thus saving some memory. Multiplication is less expensive than division, so
```
image *= 255.0/image.max()    # Uses 1 division and image.size multiplications
```
is marginally faster than
```
image /= image.max()/255.0    # Uses 1+image.size divisions
```
Since we are using basic numpy methods here, I think this is about as efficient a solution in numpy as can be. In-place operations do not change the dtype of the container array. Since the desired normalized values are floats, the audio and image arrays need to have floating-point point dtype before the in-place operations are performed. If they are not already of floating-point dtype, you'll need to convert them using astype. For example,
```
image = image.astype('float64')
```