Question:
Text[Multi-Level] Classification with many outputs Problem Statement: To classify a text document to which category it belongs and also to classify up to two levels of the category. Sample Training Set:
```
Description Category    Level1  Level2
The gun shooting that happened in Vegas killed two  Crime | High    Crime   High
Donald Trump elected as President of America    Politics | High Politics    High
Rian won in football qualifier  Sports | Low    Sports  Low
Brazil won in football final    Sports | High   Sports  High
```
Initial Attempt: I tried to create a classifier model which would try to classify the Category using Random forest method and it gave me 90% overall. Code1:
```
import pandas as pd
#import numpy as np

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import BernoulliNB
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
#from stemming.porter2 import stem

from nltk.corpus import stopwords

from sklearn.model_selection import cross_val_score

stop = stopwords.words('english')
data_file = "Training_dataset_70k"

#Reading the input/ dataset
data = pd.read_csv( data_file, header = 0, delimiter= "\t", quoting = 3, encoding = "utf8")
data = data.dropna()

#Removing stopwords, punctuation and stemming
data['Description'] = data['Description'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))
data['Description'] = data['Description'].str.replace('[^\w\s]',' ').replace('\s+',' ')
#data['Description'] = data['Description'].apply(lambda x: ' '.join([stem(word) for word in x.split()]))

train_data, test_data, train_label,  test_label = train_test_split(data.Description, data.Category, test_size=0.3, random_state=100)

RF = RandomForestClassifier(n_estimators=10)
vectorizer = TfidfVectorizer( max_features = 40000, ngram_range = ( 1,3 ), sublinear_tf = True )
data_features = vectorizer.fit_transform( train_data )
RF.fit(data_features, train_label)
test_data_feature = vectorizer.transform(test_data)
Output_predict = RF.predict(test_data_feature)
print "Overall_Accuracy: " + str(np.mean(Output_predict == test_label))
with codecs.open("out_Category.txt", "w", "utf8") as out:
    for inp, pred, act in zip(test_data, Output_predict, test_label):
        try:
            out.write("{}\t{}\t{}\n".format(inp, pred, act))
        except:
            continue
```
Problem: I want to add two more level to the model they are Level1 and Level2 the reasons for adding them is when I ran classification for Level1 alone I got 96% accuracy. I am stuck at splitting training and test dataset and to train a model which has three classifications. Is it possible to create a model with three classification or should I create three models? How to split train and test data? Edit1: import string import codecs import pandas as pd import numpy as np
```
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import BernoulliNB
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from stemming.porter2 import stem

from nltk.stem import PorterStemmer
from nltk.corpus import stopwords

from sklearn.model_selection import cross_val_score

stop = stopwords.words('english')

data_file = "Training_dataset_70k"
#Reading the input/ dataset
data = pd.read_csv( data_file, header = 0, delimiter= "\t", quoting = 3, encoding = "utf8")
data = data.dropna()
#Removing stopwords, punctuation and stemming
data['Description'] = data['Description'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))
data['Description'] = data['Description'].str.replace('[^\w\s]',' ').replace('\s+',' ')

train_data, test_data, train_label,  test_label = train_test_split(data.Description, data[["Category", "Level1", "Level2"]], test_size=0.3, random_state=100)
RF = RandomForestClassifier(n_estimators=2)
vectorizer = TfidfVectorizer( max_features = 40000, ngram_range = ( 1,3 ), sublinear_tf = True )
data_features = vectorizer.fit_transform( train_data )
print len(train_data), len(train_label)
print train_label
RF.fit(data_features, train_label)
test_data_feature = vectorizer.transform(test_data)
#print test_data_feature
Output_predict = RF.predict(test_data_feature)
print "BreadCrumb_Accuracy: " + str(np.mean(Output_predict == test_label))
with codecs.open("out_bread_crumb.txt", "w", "utf8") as out:
    for inp, pred, act in zip(test_data, Output_predict, test_label):
        try:
            out.write("{}\t{}\t{}\n".format(inp, pred, act))
        except:
            continue
```

Response:
The scikit-learn Random Forest Classifier natively supports multiple outputs (see this example). Therefore, you do not need to create three separate models. From the documentation of RandomForestClassifier.fit, the inputs to fit functions are: X : array-like or sparse matrix of shape = [n_samples, n_features] y : array-like, shape = [n_samples] or [n_samples, n_outputs] Therefore, you need an array y (your labels) of size N x 3 as your input to your RandomForestClassifier. In order to split your training and test set, you can do:
```
train_data, test_data, train_label,  test_label = train_test_split(data.Description, data[['Category','Level 1','Level 2']], test_size=0.3, random_state=100)
```
Your train_label and test_label should be arrays of size N x 3 that you can use to fit your model compare your predictions (NB: I have not tested it here, you might need to do some transposes).


Question:
ValueError: Number of features of the model must match the input I'm getting this error when trying to predict using a model I built in scikit learn. I know that there are a bunch of questions about this but mine seems different from them because I am wildly off between my input and model features. Here is my code for training my model (FYI the .csv file has 45 columns with one being the known value):
```
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn import ensemble
from sklearn.metrics import mean_absolute_error
from sklearn.externals import joblib

df = pd.read_csv("Cinderella.csv")

features_df = pd.get_dummies(df, columns=['Overall_Sentiment', 'Word_1','Word_2','Word_3','Word_4','Word_5','Word_6','Word_7','Word_8','Word_9','Word_10','Word_11','Word_1','Word_12','Word_13','Word_14','Word_15','Word_16','Word_17','Word_18','Word_19','Word_20','Word_21','Word_22','Word_23','Word_24','Word_25','Word_26','Word_27','Word_28','Word_29','Word_30','Word_31','Word_32','Word_33','Word_34','Word_35','Word_36','Word_37','Word_38','Word_39','Word_40','Word_41', 'Word_42', 'Word_43'], dummy_na=True)

del features_df['Slope']

X = features_df.as_matrix()
y = df['Slope'].as_matrix()

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

model = ensemble.GradientBoostingRegressor(
    n_estimators=500,
    learning_rate=0.01,
    max_depth=5,
    min_samples_leaf=3,
    max_features=0.1,
    loss='lad'
)

model.fit(X_train, y_train)

joblib.dump(model, 'slope_from_sentiment_model.pkl')

mse = mean_absolute_error(y_train, model.predict(X_train))

print("Training Set Mean Absolute Error: %.4f" % mse)

mse = mean_absolute_error(y_test, model.predict(X_test))
print("Test Set Mean Absolute Error: %.4f" % mse)
```
Here is my code for the actual prediction using a different .csv file (this has 44 columns because it doesn't have any values):
```
from sklearn.externals import joblib
import pandas

model = joblib.load('slope_from_sentiment_model.pkl')

df = pandas.read_csv("Slaughterhouse_copy.csv")

features_df = pandas.get_dummies(df, columns=['Overall_Sentiment','Word_1', 'Word_2', 'Word_3', 'Word_4', 'Word_5', 'Word_6', 'Word_7', 'Word_8', 'Word_9', 'Word_10', 'Word_11', 'Word_12', 'Word_13', 'Word_14', 'Word_15', 'Word_16', 'Word_17','Word_18','Word_19','Word_20','Word_21','Word_22','Word_23','Word_24','Word_25','Word_26','Word_27','Word_28','Word_29','Word_30','Word_31','Word_32','Word_33','Word_34','Word_35','Word_36','Word_37','Word_38','Word_39','Word_40','Word_41','Word_42','Word_43'], dummy_na=True)

predicted_slopes = model.predict(features_df)
```
When I run the prediction file I get:
```
ValueError: Number of features of the model must match the input. Model n_features is 146 and input n_features is 226.
```
If anyone could help me it would be greatly appreciated! Thanks in advance!

Response:
The reason you're getting the error is due to the different distinct values in your features where you're generating the dummy values with get_dummies. Let's suppose the Word_1 column in your training set has the following distinct words: the, dog, jumps, roof, off. That's 5 distinct words so pandas will generate 5 features for Word_1. Now, if your scoring dataset has a different number of distinct words in the Word_1 column, then you're going to get a different number of features. How to fix: You'll want to concatenate your training and scoring datasets using concat, apply get_dummies, and then split your datasets. That'll ensure you have captured all the distinct values in your columns. Given that you're using two different csv's, you probably want to generate a column that specifies your training vs scoring dataset. Example solution:
```
train_df = pd.read_csv("Cinderella.csv")
train_df['label'] = 'train'

score_df = pandas.read_csv("Slaughterhouse_copy.csv")
score_df['label'] = 'score'

# Concat
concat_df = pd.concat([train_df , score_df])

# Create your dummies
features_df = pd.get_dummies(concat_df, columns=['Overall_Sentiment', 'Word_1','Word_2','Word_3','Word_4','Word_5','Word_6','Word_7','Word_8','Word_9','Word_10','Word_11','Word_1','Word_12','Word_13','Word_14','Word_15','Word_16','Word_17','Word_18','Word_19','Word_20','Word_21','Word_22','Word_23','Word_24','Word_25','Word_26','Word_27','Word_28','Word_29','Word_30','Word_31','Word_32','Word_33','Word_34','Word_35','Word_36','Word_37','Word_38','Word_39','Word_40','Word_41', 'Word_42', 'Word_43'], dummy_na=True)

# Split your data
train_df = features_df[features_df['label'] == 'train']
score_df = features_df[features_df['label'] == 'score']

# Drop your labels
train_df = train_df.drop('label', axis=1)
score_df = score_df.drop('label', axis=1)

# Now delete your 'slope' feature, create your features matrix, and create your model as you have already shown in your example
...
```


Question:
Explain onehotencoder using python I am new to scikit-learn library and have been trying to play with it for prediction of stock prices. I was going through its documentation and got stuck at the part where they explain OneHotEncoder(). Here is the code that they have used :
```

```python
from sklearn.preprocessing import OneHotEncoder
```

```python
enc = OneHotEncoder()
```

```python
enc.fit([[0, 0, 3], [1, 1, 0], [0, 2, 1], [1, 0, 2]])
#Output
#OneHotEncoder(categorical_features='all', dtype=<... 'numpy.float64'>,
#       handle_unknown='error', n_values='auto', sparse=True)
```

```python
enc.n_values_
#Output
#array([2, 3, 4])
```

```python
enc.feature_indices_
#Output
#array([0, 2, 5, 9])
```

```python
enc.transform([[0, 1, 1]]).toarray()
#Output
#array([[ 1.,  0.,  0.,  1.,  0.,  0.,  1.,  0.,  0.]])
#```
#Can someone please explain it to me step by step what is happening here? I have a clear idea how One hot encoder works but I'm not able to figure out how this code works. Any help is appreciated. Thanks!
```

Response:
Lets start off first by writing down what you would expect (assuming you know what One Hot Encoding means) unecoded
```
f0 f1 f2
0, 0, 3
1, 1, 0
0, 2, 1
1, 0, 2
```
encoded
```
|f0|  |  f1 |  |   f2   |

1, 0, 1, 0, 0, 0, 0, 0, 1
0, 1, 0, 1, 0, 1, 0, 0, 0
1, 0, 0, 0, 1, 0, 1, 0, 0
0, 1, 1, 0, 0, 0, 0, 1, 0
```
To get encoded:
```
enc.fit([[0, 0, 3], [1, 1, 0], [0, 2, 1], [1, 0, 2]]),
```
if you use the default n_values='auto'. In using default='auto' you're specifying that the values your features (columns of unencoded) could possibly take on can be inferred from the values in the columns of the data handed to fit. That brings us to enc.n_values_ from the docs: Number of values per feature.
```
enc.n_values_
array([2, 3, 4])
```
The above means that f0 (column 1) can take on 2 values (0, 1), f1 can take on 3 values, (0, 1, 2) and f2 can take on 4 values (0, 1, 2, 3). Indeed these are the values from the features f1, f2 ,f3 in the unencoded feature matrix. then,
```
enc.feature_indices_
array([0, 2, 5, 9])
```
from the docs: Indices to feature ranges. Feature i in the original data is mapped to features from feature_indices_[i] to feature_indices_[i+1] (and then potentially masked by active_features_ afterwards) Given is the range of positions (in the encoded space) that features f1, f2, f3 can take on.
```
f1: [0, 1], f2: [2, 3, 4], f3: [5, 6, 7, 8]
```
Mapping the vector [0, 1, 1] into one hot encoded space (under the mapping by we got from enc.fit):
```
1, 0, 0, 1, 0, 0, 1, 0, 0
```
How? The first feature in the f0 so that maps to position 0 (if the element was 1 instead of 0 we would map it into position 1). The next element 1 maps into position 3 because f1 starts at position 2 and the element 1 is the second possible value f1 can take on. Finally the third element 1 takes on position 6 since it the second possible value f2 takes on and f2 starts getting mapped from position 5. Hope that clears up some stuff.


Question:
sklearn grid search with grouped K fold cv generator I am trying to implement a grid search over parameters in sklearn using randomized search and a grouped k fold cross-validation generator. The following works:
```
skf=StratifiedKFold(n_splits=5,shuffle=True,random_state=0)
rs=sklearn.model_selection.RandomizedSearchCV(clf,parameters,scoring='roc_auc',cv=skf,n_iter=10)
rs.fit(X,y)
```
This doesn't
```
gkf=GroupKFold(n_splits=5)
rs=sklearn.model_selection.RandomizedSearchCV(clf,parameters,scoring='roc_auc',cv=gkf,n_iter=10)
rs.fit(X,y)

#ValueError: The groups parameter should not be None
```
How do I indicate the groups parameter? Neither does this
```
gkf=GroupKFold(n_splits=5)
fv = gkf.split(X, y, groups=groups)
rs=sklearn.model_selection.RandomizedSearchCV(clf,parameters,scoring='roc_auc',cv=gkf,n_iter=10)
rs.fit(X,y)

#TypeError: object of type 'generator' has no len()
```

Response:
For reference, this is done via
```
rs.fit(X,y,groups=groups)
```
for
```
rs=sklearn.model_selection.RandomizedSearchCV(forest,parameters,scoring='roc_auc',cv=gkf,n_iter=10)
```


Question:
Difference between Shuffle and Random_State in train test split? I tried both on a small dataset sample and it returned the same output. So the question is, what is the difference between the "shuffle" and the "random_state" parameter in scikit's train-test-split method? Code for MWE:
```
X, y = np.arange(10).reshape((5, 2)), range(5)
train_test_split(y, shuffle=False)

Out: [[0, 1, 2], [3, 4]]

train_test_split(y, random_state=0)

Out: [[0, 1, 2], [3, 4]]
```

Response:
Sometimes experimenting may help understand how a function works. Say if you have a DataFrame of the sort:
```
X  Y
0  A  2
1  A  3
2  A  2
3  B  0
4  B  0
```
We'll go over the different things that you can do with the function train_test_split: if you input train, test = train_test_split(df, test_size=2/5, shuffle=False, random_state=None), you will always end up with:
```
# TRAIN
   X  Y
0  A  2
1  A  3
2  A  2

#TEST
   X  Y
3  B  0
4  B  0
```
if you input train, test = train_test_split(df, test_size=2/5, shuffle=False, random_state=1) or any other int for random_state, you will get the same:
```
# TRAIN
   X  Y
0  A  2
1  A  3
2  A  2

#TEST
   X  Y
3  B  0
4  B  0
```
This comes from the fact that you decided not to shuffle your dataset, so random_state is not used by the function. Now, if you do train, test = train_test_split(df, test_size=2/5, shuffle=True, random_state=None), you will get a dataset that looks like this:
```
# TRAIN
   X  Y
4  B  0
0  A  2
1  A  3

# TEST
   X  Y
2  A  2
3  B  0
```
Note that entries have been shuffled. But note as well that if you run your code again, results might differ. Finally, if you do train, test = train_test_split(df, test_size=2/5, shuffle=True, random_state=1) or any other int for random_state, you will get two datasets with shuffled entries as well:
```
# TRAIN
   X  Y
4  B  0
0  A  2
3  B  0

# TEST
   X  Y
2  A  2
1  A  3
```
Only, this time, if you run the code again with the same random_state, the output will always remain the same. You have set a seed, which is useful for reproducibility of the results!


Question:
How to use Dynamic Time warping with kNN in python I have a time-series dataset with two lables (0 and 1). I am using Dynamic Time Warping (DTW) as a similarity measure for classification using k-nearest neighbour (kNN) as described in these two wonderful blog posts: https://nbviewer.jupyter.org/github/markdregan/K-Nearest-Neighbors-with-Dynamic-Time-Warping/blob/master/K_Nearest_Neighbor_Dynamic_Time_Warping.ipynb http://alexminnaar.com/2014/04/16/Time-Series-Classification-and-Clustering-with-Python.html
```
Arguments
---------
n_neighbors : int, optional (default = 5)
    Number of neighbors to use by default for KNN

max_warping_window : int, optional (default = infinity)
    Maximum warping window allowed by the DTW dynamic
    programming function

subsample_step : int, optional (default = 1)
    Step size for the timeseries array. By setting subsample_step = 2,
    the timeseries length will be reduced by 50% because every second
    item is skipped. Implemented by x[:, ::subsample_step]
"""

def __init__(self, n_neighbors=5, max_warping_window=10000, subsample_step=1):
    self.n_neighbors = n_neighbors
    self.max_warping_window = max_warping_window
    self.subsample_step = subsample_step

def fit(self, x, l):
    """Fit the model using x as training data and l as class labels

    Arguments
    ---------
    x : array of shape [n_samples, n_timepoints]
        Training data set for input into KNN classifer

    l : array of shape [n_samples]
        Training labels for input into KNN classifier
    """

    self.x = x
    self.l = l

def _dtw_distance(self, ts_a, ts_b, d = lambda x,y: abs(x-y)):
    """Returns the DTW similarity distance between two 2-D
    timeseries numpy arrays.

    Arguments
    ---------
    ts_a, ts_b : array of shape [n_samples, n_timepoints]
        Two arrays containing n_samples of timeseries data
        whose DTW distance between each sample of A and B
        will be compared

    d : DistanceMetric object (default = abs(x-y))
        the distance measure used for A_i - B_j in the
        DTW dynamic programming function

    Returns
    -------
    DTW distance between A and B
    """

    # Create cost matrix via broadcasting with large int
    ts_a, ts_b = np.array(ts_a), np.array(ts_b)
    M, N = len(ts_a), len(ts_b)
    cost = sys.maxint * np.ones((M, N))

    # Initialize the first row and column
    cost[0, 0] = d(ts_a[0], ts_b[0])
    for i in xrange(1, M):
        cost[i, 0] = cost[i-1, 0] + d(ts_a[i], ts_b[0])

    for j in xrange(1, N):
        cost[0, j] = cost[0, j-1] + d(ts_a[0], ts_b[j])

    # Populate rest of cost matrix within window
    for i in xrange(1, M):
        for j in xrange(max(1, i - self.max_warping_window),
                        min(N, i + self.max_warping_window)):
            choices = cost[i - 1, j - 1], cost[i, j-1], cost[i-1, j]
            cost[i, j] = min(choices) + d(ts_a[i], ts_b[j])

    # Return DTW distance given window
    return cost[-1, -1]

def _dist_matrix(self, x, y):
    """Computes the M x N distance matrix between the training
    dataset and testing dataset (y) using the DTW distance measure

    Arguments
    ---------
    x : array of shape [n_samples, n_timepoints]

    y : array of shape [n_samples, n_timepoints]

    Returns
    -------
    Distance matrix between each item of x and y with
        shape [training_n_samples, testing_n_samples]
    """

    # Compute the distance matrix
    dm_count = 0

    # Compute condensed distance matrix (upper triangle) of pairwise dtw distances
    # when x and y are the same array
    if(np.array_equal(x, y)):
        x_s = np.shape(x)
        dm = np.zeros((x_s[0] * (x_s[0] - 1)) // 2, dtype=np.double)

        p = ProgressBar(shape(dm)[0])

        for i in xrange(0, x_s[0] - 1):
            for j in xrange(i + 1, x_s[0]):
                dm[dm_count] = self._dtw_distance(x[i, ::self.subsample_step],
                                                  y[j, ::self.subsample_step])

                dm_count += 1
                p.animate(dm_count)

        # Convert to squareform
        dm = squareform(dm)
        return dm

    # Compute full distance matrix of dtw distnces between x and y
    else:
        x_s = np.shape(x)
        y_s = np.shape(y)
        dm = np.zeros((x_s[0], y_s[0]))
        dm_size = x_s[0]*y_s[0]

        p = ProgressBar(dm_size)

        for i in xrange(0, x_s[0]):
            for j in xrange(0, y_s[0]):
                dm[i, j] = self._dtw_distance(x[i, ::self.subsample_step],
                                              y[j, ::self.subsample_step])
                # Update progress bar
                dm_count += 1
                p.animate(dm_count)

        return dm

def predict(self, x):
    """Predict the class labels or probability estimates for
    the provided data

    Arguments
    ---------
      x : array of shape [n_samples, n_timepoints]
          Array containing the testing data set to be classified

    Returns
    -------
      2 arrays representing:
          (1) the predicted class labels
          (2) the knn label count probability
    """

    dm = self._dist_matrix(x, self.x)

    # Identify the k nearest neighbors
    knn_idx = dm.argsort()[:, :self.n_neighbors]

    # Identify k nearest labels
    knn_labels = self.l[knn_idx]

    # Model Label
    mode_data = mode(knn_labels, axis=1)
    mode_label = mode_data[0]
    mode_proba = mode_data[1]/self.n_neighbors

    return mode_label.ravel(), mode_proba.ravel()
```
However, for classification with kNN the two posts use their own kNN algorithms. I want to use sklearn's options such as gridsearchcv in my classification. Therefore, I would like to know how I can use Dynamic Time Warping (DTW) with sklearn kNN. Note: I am not limited to sklearn and happy to receive answers in other libraries as well I am happy to provide more details if needed.

Response:
You can use a custom metric for KNN. Therefore you only need to implement DTW yourself (or use/adapt any existing DTW implementation in python) [gist of this code].
```
import numpy as np
from scipy.spatial import distance
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report

#toy dataset
X = np.random.random((100,10))
y = np.random.randint(0,2, (100))
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

#custom metric
def DTW(a, b):
    an = a.size
    bn = b.size
    pointwise_distance = distance.cdist(a.reshape(-1,1),b.reshape(-1,1))
    cumdist = np.matrix(np.ones((an+1,bn+1)) * np.inf)
    cumdist[0,0] = 0

    for ai in range(an):
        for bi in range(bn):
            minimum_cost = np.min([cumdist[ai, bi+1],
                                   cumdist[ai+1, bi],
                                   cumdist[ai, bi]])
            cumdist[ai+1, bi+1] = pointwise_distance[ai,bi] + minimum_cost

    return cumdist[an, bn]

#train
parameters = {'n_neighbors':[2, 4, 8]}
clf = GridSearchCV(KNeighborsClassifier(metric=DTW), parameters, cv=3, verbose=1)
clf.fit(X_train, y_train)

#evaluate
y_pred = clf.predict(X_test)
print(classification_report(y_test, y_pred))
```
Which yields
```
Fitting 3 folds for each of 3 candidates, totalling 9 fits

[Parallel(n_jobs=1)]: Done   9 out of   9 | elapsed:   29.0s finished

                         precision    recall  f1-score   support

                      0       0.57      0.89      0.70        18
                      1       0.60      0.20      0.30        15

            avg / total       0.58      0.58      0.52        33
```


Question:
Python PCA on Matrix too large to fit into memory I have a csv that is 100,000 rows x 27,000 columns that I am trying to do PCA on to produce a 100,000 rows X 300 columns matrix. The csv is 9GB large. Here is currently what I'm doing:
```
from sklearn.decomposition import PCA as RandomizedPCA
import csv
import sys
import numpy as np
import pandas as pd

dataset = sys.argv[1]
X = pd.DataFrame.from_csv(dataset)
Y = X.pop("Y_Level")
X = (X - X.mean()) / (X.max() - X.min())
Y = list(Y)
dimensions = 300
sklearn_pca = RandomizedPCA(n_components=dimensions)
X_final = sklearn_pca.fit_transform(X)
```
When I run the above code, my program is killed while doing the .from_csv in step. I've been able to get around that by spliting the csv into sets of 10,000; reading them in 1 by 1, and then calling pd.concat. This allows me to get to the normalization step (X - X.mean()).... before getting killed. Is my data just too big for my macbook air? Or is there a better way to do this. I would really love to use all the data I have for my machine learning application. If i wanted to use incremental PCA as suggested by the answer below, is this how I would do it?:
```
from sklearn.decomposition import IncrementalPCA
import csv
import sys
import numpy as np
import pandas as pd

dataset = sys.argv[1]
chunksize_ = 10000
#total_size is 100000
dimensions = 300

reader = pd.read_csv(dataset, sep = ',', chunksize = chunksize_)
sklearn_pca = IncrementalPCA(n_components=dimensions)
Y = []
for chunk in reader:
    y = chunk.pop("virginica")
    Y = Y + list(y)
    sklearn_pca.partial_fit(chunk)
X = ???
#This is were i'm stuck, how do i take my final pca and output it to X,
#the normal transform method takes in an X, which I don't have because I
#couldn't fit it into memory.
```
I can't find any good examples online.

Response:
Try to divide your data or load it by batches into script, and fit your PCA with Incremetal PCA with it's partial_fit method on every batch.
```
from sklearn.decomposition import IncrementalPCA
import csv
import sys
import numpy as np
import pandas as pd

dataset = sys.argv[1]
chunksize_ = 5 * 25000
dimensions = 300

reader = pd.read_csv(dataset, sep = ',', chunksize = chunksize_)
sklearn_pca = IncrementalPCA(n_components=dimensions)
for chunk in reader:
    y = chunk.pop("Y")
    sklearn_pca.partial_fit(chunk)

# Computed mean per feature
mean = sklearn_pca.mean_
# and stddev
stddev = np.sqrt(sklearn_pca.var_)

Xtransformed = None
for chunk in pd.read_csv(dataset, sep = ',', chunksize = chunksize_):
    y = chunk.pop("Y")
    Xchunk = sklearn_pca.transform(chunk)
    if Xtransformed == None:
        Xtransformed = Xchunk
    else:
        Xtransformed = np.vstack((Xtransformed, Xchunk))
```
Useful link


Question:
Sklearn's class "StratifiedShuffleSplit" I'm little confused about how does the class StratifiedShuffleSplit of Sklearn works. The code below is from Géron's book "Hands On Machine Learning", chapter 2, where he does a stratified sampling.
```
from sklearn.model_selection import StratifiedShuffleSplit

split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_index, test_index in split.split(housing, housing["income_cat"]):
    strat_train_set = housing.loc[train_index]
    strat_test_set = housing.loc[test_index]
```
Especially, what is been doing in split.split? Thanks!

Response:
Since you did not provide a dataset, I use sklearn sample to answer this question. Prepare dataset
```
# generate data
import numpy as np
from sklearn.model_selection import StratifiedShuffleSplit
data = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]])
group_label = np.array([0, 0, 0, 1, 1, 1])
```
This generate a dataset data, which has 6 obseravations and 2 variables. group_label has 2 value, means group 0 and group 1. In this case, group 0 contains 3 samples, same is group 1. To be general, the group size are not need to be the same. Create a StratifiedShuffleSplit object instance
```
sss = StratifiedShuffleSplit(n_splits=5, test_size=0.5, random_state=0)
sss.get_n_splits(data, group_label)
```
Out: 5 In this step, you can create a instance of StratifiedShuffleSplit, you can tell the function how to split(At random_state = 0,split data 5 times,each time 50% of data will split to test set). However, it only split data when you call it in the next step. Call the instance, and split data.
```
# the instance is actually a generater
type(sss.split(data, group_label))

# split data
for train_index, test_index in sss.split(data, group_label):
     print("n_split",,"TRAIN:", train_index, "TEST:", test_index)
     X_train, X_test = X[train_index], X[test_index]
     y_train, y_test = y[train_index], y[test_index]
```
out:
```
TRAIN: [5 2 3] TEST: [4 1 0]
TRAIN: [5 1 4] TEST: [0 2 3]
TRAIN: [5 0 2] TEST: [4 3 1]
TRAIN: [4 1 0] TEST: [2 3 5]
TRAIN: [0 5 1] TEST: [3 4 2]
```
In this step, spliter you defined in the last step will generate 5 split of data one by one. For instance, in the first split, the original data is shuffled and sample 5,2,3 is selected as train set, this is also a stratified sampling by group_label; in the second split, the data is shuffled again and sample 5,1,4 is selected as train set; etc..


Question:
Using sklearn how do I calculate the tf-idf cosine similarity between documents and a query? My goal is to input 3 queries and find out which query is most similar to a set of 5 documents. So far I have calculated the tf-idf of the documents doing the following:
```
from sklearn.feature_extraction.text import TfidfVectorizer

def get_term_frequency_inverse_data_frequency(documents):
    allDocs = []
    for document in documents:
        allDocs.append(nlp.clean_tf_idf_text(document))
    vectorizer = TfidfVectorizer()
    matrix = vectorizer.fit_transform(allDocs)
    return matrix

def get_tf_idf_query_similarity(documents, query):
    tfidf = get_term_frequency_inverse_data_frequency(documents)
```
The problem I am having is now that I have tf-idf of the documents what operations do I perform on the query so I can find the cosine similarity to the documents?

Response:
Here is my suggestion: We don't have to fit the model twice. we could reuse the same vectorizer text cleaning function can be plugged into TfidfVectorizer directly using preprocessing attribute.
```
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

vectorizer = TfidfVectorizer(preprocessor=nlp.clean_tf_idf_text)
docs_tfidf = vectorizer.fit_transform(allDocs)

def get_tf_idf_query_similarity(vectorizer, docs_tfidf, query):
    """
    vectorizer: TfIdfVectorizer model
    docs_tfidf: tfidf vectors for all docs
    query: query doc

    return: cosine similarity between query and all docs
    """
    query_tfidf = vectorizer.transform([query])
    cosineSimilarities = cosine_similarity(query_tfidf, docs_tfidf).flatten()
    return cosineSimilarities
```


Question:
Using Multiple Metric Evaluation with GridSearchCV I am attempting to use multiple metrics in GridSearchCV. My project needs multiple metrics including "accuracy" and "f1 score". However, after following the sklearn models and online posts, I can't seem to get mine to work. Here is my code:
```
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import f1_score
clf = KNeighborsClassifier()

param_grid = {'n_neighbors': range(1,30), 'algorithm': ['auto','ball_tree','kd_tree', 'brute'], 'weights': ['uniform', 'distance'],'p': range(1,5)}

#Metrics for Evualation:
met_grid= ['accuracy', 'f1'] #The metric codes from sklearn

custom_knn = GridSearchCV(clf, param_grid, scoring=met_grid, refit='accuracy', return_train_score=True)

custom_knn.fit(X_train, y_train)
y_pred = custom_knn.predict(X_test)
```
My error occurs on the custom_knn.fit(X_train,y_train). Further more, if you comment-out the scoring=met_grid, refit='accuracy', return_train_score=True, it works. Here is my error:
```
ValueError: Target is multiclass but average='binary'. Please choose another average setting.
```
Also, if you could explain multiple metric evaluation or refer me to someone who can, that would be much appreciated! Thanks

Response:
f1 is a binary classification metric. For multi-class classification, you have to use averaged f1 based on different aggregation. You can find the exhaustive list of scoring available in Sklearn here. Try this!
```
scoring = ['accuracy','f1_macro']

custom_knn = GridSearchCV(clf, param_grid, scoring=scoring,
                          refit='accuracy', return_train_score=True,cv =3)
```


Question:
How can I specify the figsize of a graphviz representation of a decision tree? I have a GraphViz representation of a decision tree trained on the iris dataset.
```
import graphviz

dot_data = tree.export_graphviz(clf, out_file=None,
                     feature_names=iris.feature_names,
                     class_names=iris.target_names,
                     filled=True, rounded=True,
                     special_characters=True)

graph = graphviz.Source(dot_data)
graph
```
I am using the above code to generate the GraphViz figure, but it creates a large graph. I want to manually control the figzise of this graph. How can I do this?

Response:
After initially writing my own function to modify the DOT source code string to add in a size attribute, I stumbled upon this section in the pydotplus.graphviz.Graph documentation: All the attributes defined in the Graphviz dot language should be supported. Attributes can be set through the dynamically generated methods:
```
set_[attribute name], i.e. set_size, set_fontname
```
You can see an example of using this below. Note the syntax when calling the function, as the DOT source code requires double quotes around the width and height. The exclamation mark means it will force the image to resize until one of the dimensions matches one of the specified dimensions, which only seems to matter if the dimensions specified are larger than the original size of the graph.
```
import pydotplus
from sklearn.datasets import load_breast_cancer
from sklearn.tree import DecisionTreeClassifier, export_graphviz

# Load in the dataset from sklearn
dataset = load_breast_cancer()
X = dataset.data
y = dataset.target
col_names = dataset.feature_names

# Create and fit the decision tree
clf_dt = DecisionTreeClassifier(criterion = 'gini', max_depth = 3)
clf_dt.fit(X_train, y_train)

# Export resulting tree to DOT source code string
dot_data = export_graphviz(clf_dt,
                                feature_names=col_names,
                                out_file=None,
                                filled=True,
                                rounded=True)

pydot_graph = pydotplus.graph_from_dot_data(dot_data)
pydot_graph.write_png('original_tree.png')
pydot_graph.set_size('"5,5!"')
pydot_graph.write_png('resized_tree.png')
```
Click on the pictures to get a sense for size since it doesn't seem to show up properly in the browser. original_tree.png: resized_tree.png: Note also that pydotplus.graphviz.Graph objects have a to_string() method which returns the DOT source code string of the tree, which can also be used with the graphviz.Source object in your question:
```
import graphviz
gvz_graph = graphviz.Source(pydot_graph.to_string())
gvz_graph
```


Question:
How are "feature_importances_" ordered in Scikit-learn's RandomForestRegressor If I run a model (called clf in this case), I get output that looks like this. How can I tie this to the feature inputs that were used to train the classifier?
```

```python
clf.feature_importances_
```

array([ 0.01621506,  0.18275428,  0.09963659,... ])
```

Response:
As mentioned in the comments, it looks like the order or feature importances is the order of the "x" input variable (which I've converted from Pandas to a Python native data structure). I use this code to generate a list of types that look like this: (feature_name, feature_importance).
```
zip(x.columns, clf.feature_importances_)
```


Question:
Regression trees or Random Forest regressor with categorical inputs I have been trying to use a categorical inpust in a regression tree (or Random Forest Regressor) but sklearn keeps returning errors and asking for numerical inputs.
```
import sklearn as sk
MODEL = sk.ensemble.RandomForestRegressor(n_estimators=100)
MODEL.fit([('a',1,2),('b',2,3),('a',3,2),('b',1,3)], [1,2.5,3,4]) # does not work
MODEL.fit([(1,1,2),(2,2,3),(1,3,2),(2,1,3)], [1,2.5,3,4]) #works

MODEL = sk.tree.DecisionTreeRegressor()
MODEL.fit([('a',1,2),('b',2,3),('a',3,2),('b',1,3)], [1,2.5,3,4]) # does not work
MODEL.fit([(1,1,2),(2,2,3),(1,3,2),(2,1,3)], [1,2.5,3,4]) #works
```
To my understanding, categorical inputs should be possible in these methods without any conversion (e.g. WOE substitution). Has anyone else had this difficulty? thanks!

Response:
scikit-learn has no dedicated representation for categorical variables (a.k.a factors in R), one possible solution is to encode the strings as int using LabelEncoder:
```
import numpy as np
from sklearn.preprocessing import LabelEncoder
from sklearn.ensemble import RandomForestRegressor

X = np.asarray([('a',1,2),('b',2,3),('a',3,2),('c',1,3)])
y = np.asarray([1,2.5,3,4])

# transform 1st column to numbers
X[:, 0] = LabelEncoder().fit_transform(X[:,0])

regressor = RandomForestRegressor(n_estimators=150, min_samples_split=2)
regressor.fit(X, y)
print(X)
print(regressor.predict(X))
```
Output:
```
[[ 0.  1.  2.]
 [ 1.  2.  3.]
 [ 0.  3.  2.]
 [ 2.  1.  3.]]
[ 1.61333333  2.13666667  2.53333333  2.95333333]
```
But remember that this is a slight hack if a and b are independent categories and it only works with tree-based estimators. Why? Because b is not really bigger than a. The correct way would be to use the OneHotEncoder after the LabelEncoder or pd.get_dummies yielding two separate, one-hot encoded columns for X[:, 0].
```
import numpy as np
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
from sklearn.ensemble import RandomForestRegressor

X = np.asarray([('a',1,2),('b',2,3),('a',3,2),('c',1,3)])
y = np.asarray([1,2.5,3,4])

# transform 1st column to numbers
import pandas as pd
X_0 = pd.get_dummies(X[:, 0]).values
X = np.column_stack([X_0, X[:, 1:]])

regressor = RandomForestRegressor(n_estimators=150, min_samples_split=2)
regressor.fit(X, y)
print(X)
print(regressor.predict(X))
```


Question: Why does the fit and the partial_fit of the sklearn LatentDirichletAllocation return different results ? What is strange is that it seems to be exactly the same code for the fit and for the partial_fit. You can see the code at the following link : https://github.com/scikit-learn/scikit-learn/blob/c957249/sklearn/decomposition/online_lda.py#L478

Response: Not exactly the same code; partial_fit uses total_samples: " total_samples : int, optional (default=1e6) Total number of documents. Only used in the partial_fit method." https://github.com/scikit-learn/scikit-learn/blob/c957249/sklearn/decomposition/online_lda.py#L184 (partial fit) https://github.com/scikit-learn/scikit-learn/blob/c957249/sklearn/decomposition/online_lda.py#L472 (fit) https://github.com/scikit-learn/scikit-learn/blob/c957249/sklearn/decomposition/online_lda.py#L510 Just in case it is of your interest: partial_fit is a good candidate to be used whenever your dataset is really really big. So, instead of running into possible memory problems you perform your fitting in smaller batches, which is called incremental learning. So, in your case you should take into account that total_samples default's value is 1000000.0. Therefore, if you don't change this number and your real number of samples is bigger then you'll get different results from the fit method and fit_partial. Or maybe it could be the case that you are using mini-batches in the fit_partial and not covering all the samples that you provide to the fit method. And even if you do this right, you could also get different results, as stated in the documentation: "the incremental learner itself may be unable to cope with new/unseen targets classes. In this case you have to pass all the possible classes to the first partial_fit call using the classes= parameter." "[...] choosing a proper algorithm is that all of them don’t put the same importance on each example over time [...]" sklearn documentation: https://scikit-learn.org/0.15/modules/scaling_strategies.html#incremental-learning


Question: What is the difference between sklearn's transform() and predict() methods? The difference between fit() and the above mentioned two methods is very distinct. fit is present in all classes of sklearn and fits an object's internal variables according to the class, be it a training model class or a preprocessor one. The difference between transform() and predict(), however, seems to be a little vague. One general rule I have seen being thrown around is that - predict() belongs to a supervised learning class and transform() belongs to an unsupervised learning class. Nevertheless, I have found some exceptions to this general rule. K-Means (an unsupervised algorithm) and PLSRegression (a supervised one) are two classes that have both of these methods. I went through the K-Means documentation and I've understood what the two different methods return; but the implementations of these two interfaces don't seem VERY concretely defined in the documentation. Like fit(), transform() can mean different things depending upon context. For a preprocessing class, fit() calculates the mean of the dataset, for example and transform applies it. For a training class, fit() acts differently and trains the model. predict() is still very self explanatory and I have only included it in the question because the different usages of transform() seems only explainable with predict() as a point of comparison. After having gone through a multitude of questions, parts of the documentation including the development manual, the glossary, and little parts of the Hands-on ML Book, this is the conclusion I have arrived at - It seems as if these interfaces are not meant to have a clear cut definition and are meant to be flexible with every class that is written. Is this correct or am I missing something?

Response: predict methods take the input features and return something that is qualitatively different, i.e. labels y (in supervised settings) or cluster memberships (in most clustering settings). transform, on the other hand, as its name already implies, returns something that is qualitatively similar to its input but expressed in a different form, i.e. scaled features, PCA features etc. And as a rule, they are usually accompanied with a respective inverse_transform method to return the input to its original form (while arguably we cannot imagine an inverse_predict method, to go from labels or cluster memberships to features). This difference is readily apparent in cases such as the ones you mention, where both predict and transform methods are available; in kmeans, for example, predict(X) will return cluster memberships, but transform(X) will return a transformation (i.e. different version) X_new of the input features.


Question:
Feature importance 'gain' in XGBoost I want to understand how the feature importance in xgboost is calculated by 'gain'. From https://towardsdatascience.com/be-careful-when-interpreting-your-features-importance-in-xgboost-6e16132588e7: ‘Gain’ is the improvement in accuracy brought by a feature to the branches it is on. The idea is that before adding a new split on a feature X to the branch there was some wrongly classified elements, after adding the split on this feature, there are two new branches, and each of these branch is more accurate (one branch saying if your observation is on this branch then it should be classified as 1, and the other branch saying the exact opposite). In scikit-learn the feature importance is calculated by the gini impurity/information gain reduction of each node after splitting using a variable, i.e. weighted impurity average of node - weighted impurity average of left child node - weighted impurity average of right child node (see also: https://stats.stackexchange.com/questions/162162/relative-variable-importance-for-boosting) I wonder if xgboost also uses this approach using information gain or accuracy as stated in the citation above. I've tried to dig in the code of xgboost and found out this method (already cut off irrelevant parts):
```
def get_score(self, fmap='', importance_type='gain'):
    trees = self.get_dump(fmap, with_stats=True)

    importance_type += '='
    fmap = {}
    gmap = {}
    for tree in trees:
        for line in tree.split('\n'):
            # look for the opening square bracket
            arr = line.split('[')
            # if no opening bracket (leaf node), ignore this line
            if len(arr) == 1:
                continue

            # look for the closing bracket, extract only info within that bracket
            fid = arr[1].split(']')

            # extract gain or cover from string after closing bracket
            g = float(fid[1].split(importance_type)[1].split(',')[0])

            # extract feature name from string before closing bracket
            fid = fid[0].split('<')[0]

            if fid not in fmap:
                # if the feature hasn't been seen yet
                fmap[fid] = 1
                gmap[fid] = g
            else:
                fmap[fid] += 1
                gmap[fid] += g

    return gmap
```
So 'gain' is extracted from dump file of each booster but how is it actually measured?

Response: Nice question. The gain is calculated using this equation: For a deep explanation read this: https://xgboost.readthedocs.io/en/latest/tutorials/model.html


Question:
Difference between model score() vs r2_score I am training a LinearRegression() classifier and trying to gauge its prediction accruacy
```
from sklearn.metrics import r2_score
from sklearn.linear_model import LinearRegression
regr_rf = LinearRegression()

regr_rf.fit(df[features],df['label'])
y_rf = regr_rf.predict(df[features])
score = regr_rf.score(df[features],df['label'])
print score
score2 = r2_score(y_rf,df['label'])
print score2
```
both score and score2 are showing very different value. I though the score function of the model is suppose to be the same as r2_score calculated explicitly

Response:
Your usage of r2_score is wrong. First argument should be true values, not the predicted values. According to the documentation:
```
r2_score(y_true, y_pred, ...)
```
So change this line score2 = r2_score(y_rf,df['label']) in your code to:
```
score2 = r2_score(df['label'], y_rf)
```
And then compare the results.


Question:
sequence matching algorithm in python I have a list of sentences such as this:
```
errList = [ 'Ragu ate lunch but didnt have Water for drinks',
            'Rams ate lunch but didnt have Gatorade for drinks',
            'Saya ate lunch but didnt have :water for drinks',
            'Raghu ate lunch but didnt have water for drinks',
            'Hanu ate lunch but didnt have -water for drinks',
            'Wayu ate lunch but didnt have water for drinks',
            'Viru ate lunch but didnt have .water 4or drinks',

            'kk ate lunch & icecream but did have Water for drinks',
            'M ate lunch &and icecream but did have Gatorade for drinks',
            'Parker ate lunch icecream but didnt have :water for drinks',
            'Sassy ate lunch and icecream but didnt have water for drinks',
            'John ate lunch and icecream but didnt have -water for drinks',
            'Pokey ate lunch and icecream but didnt have Water for drinks',
            'Laila ate lunch and icecream but did have water 4or drinks',
        ]
```
I want to find out count of longest phrases/part (phrase must be more than 2 words) of sentences in each element of list? In following example, output will look closer to this (longest phrase as key and count as value):
```
{ 'ate lunch but didnt have': 7,
  'water for drinks': 7,
  'ate lunch and icecream': 4,
  'didnt have water': 3,
  'didnt have Water': 2    # case sensitives
}
```
Using re module is out of question since problem is close to sequence matching or perhaps using nltk or perhaps scikit-learn ? I have some familiarity with NLP and scikit but not enough to solve this? If I solve this, I will publish it here.

Response:
It's not too painful with scikit-learn with a bit of numpy foo as well. A word of warning though, here I've just the defaults for preprocessing, if you're interested in the punctuation in your dataset then you will need to tweak this.
```
from sklearn.feature_extraction.text import CountVectorizer

# Find all the phrases >2 up to the max length
cv = CountVectorizer(ngram_range=(3, max([len(x.split(' ')) for x in errList])))
# Get the counts of the phrases
err_counts = cv.fit_transform(errList)
# Get the sum of each of the phrases
err_counts = err_counts.sum(axis=0)
# Mess about with the types, sparsity is annoying
err_counts = np.squeeze(np.asarray(err_counts))
# Retrieve the actual phrases that we're working with
feat_names = np.array(cv.get_feature_names())

# We don't have to sort here, but it's nice to if you want to print anything
err_counts_sorted = err_counts.argsort()[::-1]
feat_names = feat_names[err_counts_sorted]
err_counts = err_counts[err_counts_sorted]

# This is the dictionary that you were after
err_dict = dict(zip(feat_names, err_counts))
```
Here's the output for the top few
```
11 but didnt have
10 have water for drinks
10 have water for
10 water for drinks
10 but didnt have water
10 didnt have water
9 but didnt have water for drinks
9 but didnt have water for
9 didnt have water for drinks
9 didnt have water for
```


Question:
LDA ignoring n_components? When I am trying to work with LDA from Scikit-Learn, it keeps only giving me one component, even though I am asking for more:
```

```python
from sklearn.lda import LDA
```

```python
x = np.random.randn(5,5)
```

```python
y = [True, False, True, False, True]
```

```python
for i in range(1,6):
    lda = LDA(n_components=i)
    model = lda.fit(x,y)
    model.transform(x)
#Output
#```
#Gives
#```
#/Users/orthogonal/virtualenvs/osxml/lib/python2.7/site-packages/sklearn/lda.py:161: UserWarning: Variables are collinear
#  warnings.warn("Variables are collinear")
#array([[-0.12635305],
#       [-1.09293574],
#       [ 1.83978459],
#       [-0.37521856],
#       [-0.24527725]])
#array([[-0.12635305],
#       [-1.09293574],
#       [ 1.83978459],
#       [-0.37521856],
#       [-0.24527725]])
#array([[-0.12635305],
#       [-1.09293574],
#       [ 1.83978459],
#       [-0.37521856],
#       [-0.24527725]])
#array([[-0.12635305],
#       [-1.09293574],
#       [ 1.83978459],
#       [-0.37521856],
#       [-0.24527725]])
#array([[-0.12635305],
#       [-1.09293574],
#       [ 1.83978459],
#       [-0.37521856],
#       [-0.24527725]])
#```
#As you can see, it's only printing out one dimension each time. Why is this? Does it have anything to do with the variables being collinear? Additionally, when I do this with Scikit-Learn's PCA, it gives me what I want.
#```
```

```python
from sklearn.decomposition import PCA
```

```python
for i in range(1,6):
    pca = PCA(n_components=i)
    model = pca.fit(x)
    model.transform(x)

#Output
#array([[ 0.83688322],
#       [ 0.79565477],
#       [-2.4373344 ],
#       [ 0.72500848],
#       [ 0.07978792]])
#array([[ 0.83688322, -1.56459039],
#       [ 0.79565477,  0.84710518],
#       [-2.4373344 , -0.35548589],
#       [ 0.72500848, -0.49079647],
#       [ 0.07978792,  1.56376757]])
#array([[ 0.83688322, -1.56459039, -0.3353066 ],
#       [ 0.79565477,  0.84710518, -1.21454498],
#       [-2.4373344 , -0.35548589, -0.16684946],
#       [ 0.72500848, -0.49079647,  1.09006296],
#       [ 0.07978792,  1.56376757,  0.62663807]])
#array([[ 0.83688322, -1.56459039, -0.3353066 ,  0.22196922],
#       [ 0.79565477,  0.84710518, -1.21454498, -0.15961993],
#       [-2.4373344 , -0.35548589, -0.16684946, -0.04114339],
#       [ 0.72500848, -0.49079647,  1.09006296, -0.2438673 ],
#       [ 0.07978792,  1.56376757,  0.62663807,  0.2226614 ]])
#array([[  8.36883220e-01,  -1.56459039e+00,  -3.35306597e-01,
#          2.21969223e-01,  -1.66533454e-16],
#       [  7.95654771e-01,   8.47105182e-01,  -1.21454498e+00,
#         -1.59619933e-01,   3.33066907e-16],
#       [ -2.43733440e+00,  -3.55485895e-01,  -1.66849458e-01,
#         -4.11433949e-02,   0.00000000e+00],
#       [  7.25008484e-01,  -4.90796471e-01,   1.09006296e+00,
#         -2.43867297e-01,  -1.38777878e-16],
#       [  7.97879229e-02,   1.56376757e+00,   6.26638070e-01,
#          2.22661402e-01,   2.22044605e-16]])
#```
```

Response: This is the relevant, dimension-reducing line of LDA.transform, it uses scalings_. As described in the docstring, scalings_ has maximally n_classes - 1 columns. This is then the maximal number of columns you can hope to obtain using transform. In your case, 2 classes (True, False), yields maximally 1 column.


Question: Predict with sklearn-KNN using median (instead of mean) Sklearn-KNN allows one to set weights (e.g., uniform, distance) when calculating the mean x nearest neighbours. Instead of predicting with the mean, is it possible to predict with the median (perhaps with a user-defined function)?

Response:
There is no built-in parameter to adjust the weighting to use the median rather than the mean (you can see in the source that the mean is hard-coded). But because scikit-learn estimators are just Python classes, you can subclass KNeighborsRegressor and override the predict method to do whatever you want. Here's a quick example, where I've copied and pasted the original predict() method and modified the relevant piece:
```
from sklearn.neighbors.regression import KNeighborsRegressor, check_array, _get_weights

class MedianKNNRegressor(KNeighborsRegressor):
    def predict(self, X):
        X = check_array(X, accept_sparse='csr')

        neigh_dist, neigh_ind = self.kneighbors(X)

        weights = _get_weights(neigh_dist, self.weights)

        _y = self._y
        if _y.ndim == 1:
            _y = _y.reshape((-1, 1))

        ######## Begin modification
        if weights is None:
            y_pred = np.median(_y[neigh_ind], axis=1)
        else:
            # y_pred = weighted_median(_y[neigh_ind], weights, axis=1)
            raise NotImplementedError("weighted median")
        ######### End modification

        if self._y.ndim == 1:
            y_pred = y_pred.ravel()

        return y_pred

X = np.random.rand(100, 1)
y = 20 * X.ravel() + np.random.rand(100)
clf = MedianKNNRegressor().fit(X, y)
print(clf.predict(X[:5]))
# [  2.38172861  13.3871126    9.6737255    2.77561858  17.07392584]
```
I've left out the weighted version, because I don't know of a simple way to compute a weighted median with numpy/scipy, but it would be straightforward to add in once that function is available.


Question:
OneHotEncoder - encoding only some of categorical variable columns Let's assume that I have a pandas dataframe with the following column names: 'age' (e.g. 33, 26, 51 etc) 'seniority' (e.g. 'junior', 'senior' etc) 'gender' (e.g. 'male', 'female') 'salary' (e.g. 32000, 40000, 64000 etc) I want to transform the seniority categorical variables to one hot encoded values. For this reason I am doing the following:
```
from sklearn.preprocessing import LabelEncoder
label_encoder = LabelEncoder()
data['seniority'] = label_encoder.fit_transform(data['seniority'])

from sklearn.preprocessing import OneHotEncoder
one_hot_encoder = OneHotEncoder(categorical_features=[1])
data = one_hot_encoder.fit_transform(data.values)
```
But then I am getting this error
```
ValueError: could not convert string to float: 'gender'
```
at line
```
data = one_hot_encoder.fit_transform(data.values)
```
However, I have explicitly specified that categorical_features=[1] so only column 1 (seniority) should be considered for this one hot encoding. How can I fix this error (except for example by dropping the column 'gender')? I was using pandas.get_dummies in the past and I did not have this problem.

Response:
I think for this case you should stick to pd.get_dummies:
```

```python
data
#Output
#   age seniority  gender  salary
#0    1    junior    male       5
#1    2    senior  female       6
#2    3    junior  female       7
```

# One hot encode with get_dummies
data = pd.concat((data,pd.get_dummies(data.seniority)),1)

```python
data
#Output
#   age seniority  gender  salary  junior  senior
#0    1    junior    male       5       1       0
#1    2    senior  female       6       0       1
#2    3    junior  female       7       1       0
#```
#The problem is that sklearn's OneHotEncoder needs to have an array of ints as input. But in the array data.values, you still have the string representation of gender. You could, if you wanted, just one hot encode the seniority values, but if you want to know the meaning of those features, it's not very nice, you have to pass it the column names manually (which is unfeasible in a lot of cases):
#```
#from sklearn.preprocessing import LabelEncoder
#label_encoder = LabelEncoder()
#data['seniority'] = label_encoder.fit_transform(data['seniority'])
```

from sklearn.preprocessing import OneHotEncoder
one_hot_encoder = OneHotEncoder(sparse=False)
data[['junior','senior']] = one_hot_encoder.fit_transform(data['seniority'].values.reshape(-1,1))

```python
data
#Output
#   age  seniority  gender  salary  junior  senior
#0    1          0    male       5     1.0     0.0
#1    2          1  female       6     0.0     1.0
#2    3          0  female       7     1.0     0.0
#```
#Or, if the feature names don't matter:
#```
#from sklearn.preprocessing import LabelEncoder
#label_encoder = LabelEncoder()
#data['seniority'] = label_encoder.fit_transform(data['seniority'])
```

from sklearn.preprocessing import OneHotEncoder
one_hot_encoder = OneHotEncoder(sparse=False)
data = pd.concat((data,pd.DataFrame(one_hot_encoder.fit_transform(data['seniority'].values.reshape(-1,1)))),1)

   age  seniority  gender  salary    0    1
0    1          0    male       5  1.0  0.0
1    2          1  female       6  0.0  1.0
2    3          0  female       7  1.0  0.0
```
But in the end, pd.get_dummies does the job in a much nicer way (IMO)


Question:
Gaussian Mixture Models of an Image's Histogram I am attempting to do automatic image segmentation of the different regions of a 2D MR image based on pixel intensity values. The first step is implementing a Gaussian Mixture Model on the image's histogram. I need to plot the resulting gaussian obtained from the score_samples method onto the histogram. I have tried following the code in the answer to (Understanding Gaussian Mixture Models). However, the resulting gaussian fails to match the histogram at all. How do I get the gaussian to match the histogram?
```
import numpy as np
import cv2
import matplotlib.pyplot as plt
from sklearn.mixture import GaussianMixture

# Read image
img = cv2.imread("test.jpg",0)

hist = cv2.calcHist([img],[0],None,[256],[0,256])
hist[0] = 0     # Removes background pixels

# Fit GMM
gmm = GaussianMixture(n_components = 3)
gmm = gmm.fit(hist)

# Evaluate GMM
gmm_x = np.linspace(0,255,256)
gmm_y = np.exp(gmm.score_samples(gmm_x.reshape(-1,1)))

# Plot histograms and gaussian curves
fig, ax = plt.subplots()
ax.hist(img.ravel(),255,[1,256])
ax.plot(gmm_x, gmm_y, color="crimson", lw=4, label="GMM")

ax.set_ylabel("Frequency")
ax.set_xlabel("Pixel Intensity")

plt.legend()

plt.show()
```
I also attempted manually constructing the gaussians with sums.
```
import numpy as np
import cv2
import matplotlib.pyplot as plt
from sklearn.mixture import GaussianMixture

def gauss_function(x, amp, x0, sigma):
    return amp * np.exp(-(x - x0) ** 2. / (2. * sigma ** 2.))

# Read image
img = cv2.imread("test.jpg",0)

hist = cv2.calcHist([img],[0],None,[256],[0,256])
hist[0] = 0     # Removes background pixels

# Fit GMM
gmm = GaussianMixture(n_components = 3)
gmm = gmm.fit(hist)

# Evaluate GMM
gmm_x = np.linspace(0,255,256)
gmm_y = np.exp(gmm.score_samples(gmm_x.reshape(-1,1)))

# Construct function manually as sum of gaussians
gmm_y_sum = np.full_like(gmm_x, fill_value=0, dtype=np.float32)
for m, c, w in zip(gmm.means_.ravel(), gmm.covariances_.ravel(), gmm.weights_.ravel()):
    gauss = gauss_function(x=gmm_x, amp=1, x0=m, sigma=np.sqrt(c))
    gmm_y_sum += gauss / np.trapz(gauss, gmm_x) * w

# Plot histograms and gaussian curves
fig, ax = plt.subplots()
ax.hist(img.ravel(),255,[1,256])
ax.plot(gmm_x, gmm_y, color="crimson", lw=4, label="GMM")
ax.plot(gmm_x, gmm_y_sum, color="black", lw=4, label="Gauss_sum", linestyle="dashed")

ax.set_ylabel("Frequency")
ax.set_xlabel("Pixel Intensity")

plt.legend()

plt.show()
```
With ax.hist(img.ravel(),255,[1,256], normed=True)

Response:
The issue was with passing the histogram rather than the array of pixel intensities to GaussianMixture.fit gmm = gmm.fit(hist). I also found that a minimum of n_components = 6 is needed to visually fit this particular histogram.
```
import numpy as np
import cv2
import matplotlib.pyplot as plt
from sklearn.mixture import GaussianMixture

# Read image
img = cv2.imread("test.jpg",0)

hist = cv2.calcHist([img],[0],None,[256],[0,256])
hist[0] = 0     # Removes background pixels

data = img.ravel()
data = data[data != 0]
data = data[data != 1]  #Removes background pixels (intensities 0 and 1)

# Fit GMM
gmm = GaussianMixture(n_components = 6)
gmm = gmm.fit(X=np.expand_dims(data,1))

# Evaluate GMM
gmm_x = np.linspace(0,253,256)
gmm_y = np.exp(gmm.score_samples(gmm_x.reshape(-1,1)))

# Plot histograms and gaussian curves
fig, ax = plt.subplots()
ax.hist(img.ravel(),255,[2,256], normed=True)
ax.plot(gmm_x, gmm_y, color="crimson", lw=4, label="GMM")

ax.set_ylabel("Frequency")
ax.set_xlabel("Pixel Intensity")

plt.legend()

plt.show()
```


Question:
Cosine similarity TSNE in sklearn.manifold I have a small problem to perform TSNE on my dataset, using cosine similarity. I have calculated the cosine similarity of all of my vectors, so I have a square matrix which contains my cosine similarity :
```
A = [[  1    0.7   0.5   0.6  ]
     [  0.7   1    0.3   0.4  ]
     [  0.5  0.3    1    0.1  ]
     [  0.6  0.4   0.1    1   ]]
```
Then, I'm using TSNE like that :
```
A = np.matrix([[1, 0.7,0.5,0.6],[0.7,1,0.3,0.4],[0.5,0.3,1,0.1],[0.6,0.4,0.1,1]])
model = manifold.TSNE(metric="precomputed")
Y = model.fit_transform(A)
```
But I'm not sure that to use precomputed metric keep the sense of my cosine similarity:
```
#[documentation][1]
If metric is “precomputed”, X is assumed to be a distance matrix
```
But when I try to use cosine metric, I got an error :
```
A = np.matrix([[1, 0.7,0.5,0.6],[0.7,1,0.3,0.4],[0.5,0.3,1,0.1],[0.6,0.4,0.1,1]])
model = manifold.TSNE(metric="cosine")
Y = model.fit_transform(A)

raise ValueError("All distances should be positive, either "
ValueError: All distances should be positive, either the metric or
precomputed distances given as X are not correct
```
So my question is, How is it possible to perform TSNE using cosine metric on an existent dataset (similarity matrix) ?

Response:
I can answer the majority of your question, however I'm not quite sure why that error is popping up in your second example. You have calculated the cosine similarity of each of your vectors, but scikit assumes a distance matrix for the input to TSNE. However this is a really simple transformation distance = 1 - similarity. So for your example
```
import numpy as np
from sklearn import manifold
A = np.matrix([[1, 0.7,0.5,0.6],[0.7,1,0.3,0.4],[0.5,0.3,1,0.1],[0.6,0.4,0.1,1]])
A = 1.-A
model = manifold.TSNE(metric="precomputed")
Y = model.fit_transform(A)
```
This should give you the transformation you want.


Question:
How to fix ValueError: multiclass format is not supported [duplicate] This question already has answers here: ValueError: continuous format is not supported (3 answers) Closed 1 year ago. This is my code and I try to calculate ROC score but I have a problem with ValueError: multiclass format is not supported. I'm already looking sci-kit learn but it doesn't help. In the end, I'm still have ValueError: multiclass format is not supported. This is my code
```
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import BaggingClassifier
from sklearn.metrics import confusion_matrix,zero_one_loss
from sklearn.metrics import classification_report,matthews_corrcoef,accuracy_score
from sklearn.metrics import roc_auc_score, auc

dtc = DecisionTreeClassifier()
bc = BaggingClassifier(base_estimator=dtc, n_estimators=10, random_state=17)
bc.fit(train_x, train_Y)
pred_y = bc.predict(test_x)

fprate, tprate, thresholds = roc_curve(test_Y, pred_y)
results = confusion_matrix(test_Y, pred_y)
error = zero_one_loss(test_Y, pred_y)
roc_auc_score(test_Y, pred_y)

FP = results.sum(axis=0) - np.diag(results)
FN = results.sum(axis=1) - np.diag(results)
TP = np.diag(results)
TN = results.sum() - (FP + FN + TP)

print('\n Time Processing: \n',time.process_time())
print('\n Confussion Matrix: \n', results)
print('\n Zero-one classification loss: \n', error)
print('\n True Positive: \n', TP)
print('\n True Negative: \n', TN)
print('\n False Positive: \n', FP)
print('\n False Negative: \n', FN)
print ('\n The Classification report:\n',classification_report(test_Y,pred_y, digits=6))
print ('MCC:', matthews_corrcoef(test_Y,pred_y))
print ('Accuracy:', accuracy_score(test_Y,pred_y))
print (auc(fprate, tprate))
print ('ROC Score:', roc_auc_score(test_Y,pred_y))
```
This is the traceback

Response:
From the docs, roc_curve: "Note: this implementation is restricted to the binary classification task." Are your label classes (y) either 1 or 0? If not, I think you have to add the pos_label parameter to your roc_curve call.
```
fprate, tprate, thresholds = roc_curve(test_Y, pred_y, pos_label='your_label')
```
Or:
```
test_Y = your_test_y_array  # these are either 1's or 0's
fprate, tprate, thresholds = roc_curve(test_Y, pred_y)
```


Question:
How does sp_randint work? I am doing hyperparameter optimisation of Random Forest classifier. I am planning to use RandomSearchCV. So by checking the available code in Scikit learn: What does sp_randint do? Does it randomly take a value from 1 to 11? Can it be replaced by other function?
```
from scipy.stats import randint as sp_randint

    param_dist = {"n_estimators": sp_randint (1, 11),
                  "max_depth": [3, None],
                  "max_features": sp_randint(1, 11),
                  "min_samples_split": sp_randint(1, 11),
                  "min_samples_leaf": sp_randint(1, 11),
                 }
```
Thank you.

Response:
sklearn.grid_search.RandomizedSearchCV can get a param_distributions parameter, mapping parameters to random distributions supporting the rvs method. In your example, this object will return random integers in the range $[1, 11)$:
```

```python
g = sp_randint(1, 11)
```

```python
g.rvs(20)
#Output
#array([ 5,  2,  9, 10,  6,  9,  9,  8,  1,  5,  1,  8,  1,  5,  5,  4,  6,
#        5,  8,  4])
#```
#You can change it to any other object meaningfully supporting the rvs method, or even a list. For example:
#```
#param_dist = {"n_estimators": [1, 3, 4],
#              "max_depth": [3, None],
#              "max_features": [1, 3, 4],
#              "min_samples_split": [1, 3, 4],
#              "min_samples_leaf": [1, 3, 4],
#             }
#```
#will work as well.
```


Question:
Distances between rankings I have two methods that rank a list of strings differently, and what we can consider to be the "right" ranking of the list (i.e. a gold standard). In other words:
```
ranked_list_of_strings_1 = method_1(list_of_strings)
 ranked_list_of_strings_2 = method_2(list_of_strings)
 correctly_ranked_list_of_strings # Some permutation of list_of_strings
```
How can I determine which method is better considering that method_1 and method_2 are black boxes? Are there any methods to measure this available either in SciPy or scikit-learn or similar libraries? In my specific case, I actually have a dataframe, and each method outputs a score. What matters is not the difference in score between the methods and the true scores, but that the methods get the ranking right (higher score means higher ranking for all columns).
```
strings        scores_method_1   scores_method_2   true_scores
5714  aeSeOg                    0.54               0.1           0.8
5741  NQXACs                    0.15               0.3           0.4
5768  zsFZQi                    0.57               0.7           0.2
```

Response: You're looking for Normalized Discounted Cumulative Gain (NDGC). It's a metric commonly used in search engine rankings to test the quality of the result ranking. The idea is that you test your ranking (in your case the two methods) against user feedback through clicks (in your cast the true rank). NDGC will tell you the quality of your ranking relative to the truth. Python has RankEval based module that implements this metric (and some others if you want to try them). The repo is here and there is a nice IPython NB with examples


Question: Class weights vs under/oversampling In imbalanced classification (with scikit-learn) what would be the difference of balancing classes (i.e. set class_weight to balanced) to oversampling with SMOTE for example? What would be the expected effects of one vs the other?

Response: Class weights directly modify the loss function by giving more (or less) penalty to the classes with more (or less) weight. In effect, one is basically sacrificing some ability to predict the lower weight class (the majority class for unbalanced datasets) by purposely biasing the model to favor more accurate predictions of the higher weighted class (the minority class). Oversampling and undersampling methods essentially give more weight to particular classes as well (duplicating observations duplicates the penalty for those particular observations, giving them more influence in the model fit), but due to data splitting that typically takes place in training this will yield slightly different results as well. Please refer to https://datascience.stackexchange.com/questions/52627/why-class-weight-is-outperforming-oversampling


Question:
Feature Hashing on multiple categorical features(columns) I would like to hash feature ‘Genre’ into 6 columns and separately feature ‘Publisher’ into another six columns. I want something like below:
```
Genre      Publisher  0    1    2    3    4    5      0    1    2    3    4    5
0     Platform  Nintendo  0.0  2.0  2.0 -1.0  1.0  0.0    0.0  2.0  2.0 -1.0  1.0  0.0
1       Racing      Noir -1.0  0.0  0.0  0.0  0.0 -1.0   -1.0  0.0  0.0  0.0  0.0 -1.0
2       Sports     Laura -2.0  2.0  0.0 -2.0  0.0  0.0   -2.0  2.0  0.0 -2.0  0.0  0.0
3  Roleplaying      John -2.0  2.0  2.0  0.0  1.0  0.0   -2.0  2.0  2.0  0.0  1.0  0.0
4       Puzzle      John  0.0  1.0  1.0 -2.0  1.0 -1.0    0.0  1.0  1.0 -2.0  1.0 -1.0
5     Platform      Noir  0.0  2.0  2.0 -1.0  1.0  0.0    0.0  2.0  2.0 -1.0  1.0  0.0
```
The following code does what I want to do
```
import pandas as pd
d = {'Genre': ['Platform', 'Racing','Sports','Roleplaying','Puzzle','Platform'], 'Publisher': ['Nintendo', 'Noir','Laura','John','John','Noir']}
df = pd.DataFrame(data=d)
from sklearn.feature_extraction import FeatureHasher
fh1 = FeatureHasher(n_features=6, input_type='string')
fh2 = FeatureHasher(n_features=6, input_type='string')
hashed_features1 = fh.fit_transform(df['Genre'])
hashed_features2 = fh.fit_transform(df['Publisher'])
hashed_features1 = hashed_features1.toarray()
hashed_features2 = hashed_features2.toarray()
pd.concat([df[['Genre', 'Publisher']], pd.DataFrame(hashed_features1),pd.DataFrame(hashed_features2)],
          axis=1)
```
This works for the above two feature but If I have lets say 40 categorical features then this approach would be tedious. Is there any other way to do?

Response:
Hashing (Update) Assuming that new categories might show up in some of the features, hashing is the way to go. Just 2 notes: Be aware of the possibility of collision and adjust the number of features accordingly In your case, you want to hash each feature separately One Hot Vector In case the number of categories for each feature is fixed and not too large, use one hot encoding. I would recommend using either of the two: sklearn.preprocessing.OneHotEncoder pandas.get_dummies Example
```
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.feature_extraction import FeatureHasher
from sklearn.preprocessing import OneHotEncoder

df = pd.DataFrame({'feature_1': ['A', 'G', 'T', 'A'],
                   'feature_2': ['cat', 'dog', 'elephant', 'zebra']})

# Approach 0 (Hashing per feature)
n_orig_features = df.shape[1]
hash_vector_size = 6
ct = ColumnTransformer([(f't_{i}', FeatureHasher(n_features=hash_vector_size,
                        input_type='string'), i) for i in range(n_orig_features)])

res_0 = ct.fit_transform(df)  # res_0.shape[1] = n_orig_features * hash_vector_size

# Approach 1 (OHV)
res_1 = pd.get_dummies(df)

# Approach 2 (OHV)
res_2 = OneHotEncoder(sparse=False).fit_transform(df)
```
res_0 :
```
array([[ 0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  1., -1.,  0., -1.],
       [ 0.,  0.,  0.,  1.,  0.,  0.,  0.,  2., -1.,  0.,  0.,  0.],
       [ 0., -1.,  0.,  0.,  0.,  0., -2.,  2.,  2., -1.,  0., -1.],
       [ 0.,  0.,  0.,  0.,  1.,  0.,  0.,  2.,  1., -1.,  0., -1.]])
```
res_1 :
```
feature_1_A  feature_1_G  feature_1_T  feature_2_cat  feature_2_dog  feature_2_elephant  feature_2_zebra
0            1            0            0              1              0                   0                0
1            0            1            0              0              1                   0                0
2            0            0            1              0              0                   1                0
3            1            0            0              0              0                   0                1
```
res_2 :
```
array([[1., 0., 0., 1., 0., 0., 0.],
       [0., 1., 0., 0., 1., 0., 0.],
       [0., 0., 1., 0., 0., 1., 0.],
       [1., 0., 0., 0., 0., 0., 1.]])
```


Question:
Python - generate array of specific autocorrelation I am interested in generating an array(or numpy Series) of length N that will exhibit specific autocorrelation at lag 1. Ideally, I want to specify the mean and variance, as well, and have the data drawn from (multi)normal distribution. But most importantly, I want to specify the autocorrelation. How do I do this with numpy, or scikit-learn? Just to be explicit and precise, this is the autocorrelation I want to control:
```
numpy.corrcoef(x[0:len(x) - 1], x[1:])[0][1]
```

Response:
If you are interested only in the auto-correlation at lag one, you can generate an auto-regressive process of order one with the parameter equal to the desired auto-correlation; this property is mentioned on the Wikipedia page, but it's not hard to prove it. Here is some sample code:
```
import numpy as np

def sample_signal(n_samples, corr, mu=0, sigma=1):
    assert -1 < corr < 1, "Auto-correlation coefficient must be between -1 and 1"

    # Find out the offset `c` and the std of the white noise `sigma_e`
    # that produce a signal with the desired mean and variance.
    # See https://en.wikipedia.org/wiki/Autoregressive_model
    # under section "Example: An AR(1) process".
    c = mu * (1 - corr)
    sigma_e = np.sqrt((sigma ** 2) * (1 - corr ** 2))

    # Sample the auto-regressive process.
    signal = [np.random.normal(mu, sigma)]
    for _ in range(1, n_samples):
        signal.append(c + corr * signal[-1] + np.random.normal(0, sigma_e))

    return np.array(signal)

def compute_corr_lag_1(signal):
    return np.corrcoef(signal[:-1], signal[1:])[0][1]

# Examples.
print(compute_corr_lag_1(sample_signal(5000, 0.5)))
print(np.mean(sample_signal(5000, 0.5, mu=2)))
print(np.std(sample_signal(5000, 0.5, sigma=3)))
```
The parameter corr lets you set the desired auto-correlation at lag one and the optional parameters, mu and sigma, let you control the mean and standard deviation of the generated signal.


Question:
How to change colors for decision tree plot using sklearn plot_tree? How to change colors in decision tree plot using sklearn.tree.plot_tree without using graphviz as in this question: Changing colors for decision tree plot created using export graphviz?
```
plt.figure(figsize=[21, 6])
ax1 = plt.subplot(121)
ax2 = plt.subplot(122)

ax1.plot(X[:, 0][y == 0], X[:, 1][y == 0], "bo")
ax1.plot(X[:, 0][y == 1], X[:, 1][y == 1], "g^")
ax1.contourf(xx, yy, pred.reshape(xx.shape), cmap=matplotlib.colors.ListedColormap(['b', 'g']), alpha=0.25)
ax1.set_title(title)

plot_tree(tree_clf, feature_names=["X", "y"], class_names=["blue", "green"], filled=True, rounded=True)
```

Response:
Many matplotlib functions follow the color cycler to assign default colors, but that doesn't seem to apply here. The following approach loops through the generated annotation texts (artists) and the clf tree structure to assign colors depending on the majority class and the impurity (gini). Note that we can't use alpha, as a transparent background would show parts of arrows that are usually hidden.
```py
from matplotlib import pyplot as plt
from matplotlib.colors import ListedColormap, to_rgb
import numpy as np
from sklearn import tree

X = np.random.rand(50, 2) * np.r_[100, 50]
y = X[:, 0] - X[:, 1] > 20

clf = tree.DecisionTreeClassifier(random_state=2021)
clf = clf.fit(X, y)

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=[21, 6])

colors = ['crimson', 'dodgerblue']
ax1.plot(X[:, 0][y == 0], X[:, 1][y == 0], "o", color=colors[0])
ax1.plot(X[:, 0][y == 1], X[:, 1][y == 1], "^", color=colors[1])
xx, yy = np.meshgrid(np.linspace(X[:, 0].min(), X[:, 0].max(), 100), np.linspace(X[:, 1].min(), X[:, 1].max(), 100))
pred = clf.predict(np.c_[(xx.ravel(), yy.ravel())])
ax1.contourf(xx, yy, pred.reshape(xx.shape), cmap=ListedColormap(colors), alpha=0.25)

# ax2.set_prop_cycle(mpl.cycler(color=colors)) # doesn't seem to work

artists = tree.plot_tree(clf, feature_names=["X", "y"], class_names=colors,
                         filled=True, rounded=True, ax=ax2)
for artist, impurity, value in zip(artists, clf.tree_.impurity, clf.tree_.value):
    # let the max value decide the color; whiten the color depending on impurity (gini)
    r, g, b = to_rgb(colors[np.argmax(value)])
    f = impurity * 2 # for N colors: f = impurity * N/(N-1) if N>1 else 0
    artist.get_bbox_patch().set_facecolor((f + (1-f)*r, f + (1-f)*g, f + (1-f)*b))
    artist.get_bbox_patch().set_edgecolor('black')

plt.tight_layout()
plt.show()
```


Question:
How to get the centroids in DBSCAN sklearn? I am using DBSCAN for clustering. However, now I want to pick a point from each cluster that represents it, but I realized that DBSCAN does not have centroids as in kmeans. However, I observed that DBSCAN has something called core points. I am thinking if it is possible to use these core points or any other alternative to obtain a representative point from each cluster. I have mentioned below the code that I have used.
```
import numpy as np
from math import pi
from sklearn.cluster import DBSCAN

#points containing time value in minutes
points = [100, 200, 600, 659, 700]

def convert_to_radian(x):
    return((x / (24 * 60)) * 2 * pi)

rad_function = np.vectorize(convert_to_radian)
points_rad = rad_function(points)

#generate distance matrix from each point
dist = points_rad[None,:] - points_rad[:, None]

#Assign shortest distances from each point
dist[((dist > pi) & (dist <= (2*pi)))] = dist[((dist > pi) & (dist <= (2*pi)))] -(2*pi)
dist[((dist > (-2*pi)) & (dist <= (-1*pi)))] = dist[((dist > (-2*pi)) & (dist <= (-1*pi)))] + (2*pi)
dist = abs(dist)

#check dist
print(dist)

#using default values, set metric to 'precomputed'
db = DBSCAN(eps=((100 / (24*60)) * 2 * pi ), min_samples = 2, metric='precomputed')

#check db
print(db)

db.fit(dist)

#get labels
labels = db.labels_

#get number of clusters
no_clusters = len(set(labels)) - (1 if -1 in labels else 0)

print('No of clusters:', no_clusters)
print('Cluster 0 : ', np.nonzero(labels == 0)[0])
print('Cluster 1 : ', np.nonzero(labels == 1)[0])

print(db.core_sample_indices_)
```
I am happy to provide more details if needed.

Response:
Why don't you estimate the centroids of the resulted estimated clusters?
```
points_of_cluster_0 = dist[labels==0,:]
centroid_of_cluster_0 = np.mean(points_of_cluster_0, axis=0)
print(centroid_of_cluster_0)

points_of_cluster_1 = dist[labels==1,:]
centroid_of_cluster_1 = np.mean(points_of_cluster_1, axis=0)
print(centroid_of_cluster_1)
```


Question:
Multiple output regression or classifier with one (or more) parameters with Python I wrote a simple linear regression and decision tree classifier code with Python's Scikit-learn library for predicting the outcome. It works well. My question is, Is there a way to do this backwards, to predict the best combination of parameter values based on imputed outcome (parameters, where accuracy will be the best). Or I can ask like this, is there a classification, regression or some other type of algorithm (Decision tree, SVM, KNN, Logistic regression, Linear regression, Polynomial regression...) that can predict multiple outcomes based on one (or more) parameter/s? I have tried to do this with putting multivariate outcome, but it shows the error: ValueError: Expected 2D array, got 1D array instead: array=[101 905 182 268 646 624 465]. Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample. This is the code that I wrote for regression:
```
import pandas as pd
from sklearn import linear_model
from sklearn import tree

dic = {'par_1': [10, 30, 13, 19, 25, 33, 23],
       'par_2': [1, 3, 1, 2, 3, 3, 2],
       'outcome': [101, 905, 182, 268, 646, 624, 465]}

df = pd.DataFrame(dic)

variables = df.iloc[:,:-1]
results = df.iloc[:,-1]

regression = linear_model.LinearRegression()
regression.fit(variables, results)

input_values = [14, 2]

prediction = regression.predict([input_values])
prediction = round(prediction[0], 2)
print(prediction)
```
This is the code that I wrote for decision tree:
```
dic = {'par_1': [10, 30, 13, 19, 25, 33, 23],
       'par_2': [1, 3, 1, 2, 3, 3, 2],
       'outcome': ['yes', 'yes', 'no', 'yes', 'no', 'no', 'yes']}

df = pd.DataFrame(dic)

variables = df.iloc[:,:-1]
results = df.iloc[:,-1]

decision_tree = tree.DecisionTreeClassifier()
decision_tree.fit(variables, results)

input_values = [18, 2]

prediction = decision_tree.predict([input_values])[0]
print(prediction)
```

Response:
As mentioned by @Justas, if you want to find the best combination of input values for which the output variable would be max/min, then it is a optimization problem. There are quite a good range of non-linear optimizers available in scipy or you can go for meta-heuristics such Genetic Algorithm, Memetic algorithm, etc. On the other hand, if your aim is to learn the inverse function, which maps the output variable into a set of input variables then the go for MultiOuputRegresssor or MultiOutputClassifier. Both of them can be used as a wrapper on top of any base estimators such as linearRegression, LogisticRegresssion, KNN, DecisionTree, SVM, etc. Example:
```
import pandas as pd
from sklearn.multioutput import MultiOutputRegressor, RegressorChain
from sklearn.linear_model import LinearRegression

dic = {'par_1': [10, 30, 13, 19, 25, 33, 23],
       'par_2': [1, 3, 1, 2, 3, 3, 2],
       'outcome': [101, 905, 182, 268, 646, 624, 465]}

df = pd.DataFrame(dic)

variables = df.iloc[:,:-1]
results = df.iloc[:,-1]

multi_output_reg = MultiOutputRegressor(LinearRegression())
multi_output_reg.fit(results.values.reshape(-1, 1),variables)

multi_output_reg.predict([[100]])

# array([[12.43124217,  1.12571947]])
# sounds sensible according to the training data

#if input variables needs to be treated as categories,
# go for multiOutputClassifier
from sklearn.multioutput import MultiOutputClassifier
from sklearn.linear_model import LogisticRegression

multi_output_clf = MultiOutputClassifier(LogisticRegression(solver='lbfgs'))
multi_output_clf.fit(results.values.reshape(-1, 1),variables)

multi_output_clf.predict([[100]])

# array([[10,  1]])
```
In most situations, finding one of the input variable value can help in predicting other variables. This approach can be achieved by ClassifierChain or RegressorChain. To understand the advantage of ClassifierChain, please refer to this example. Update:
```
dic = {'par_1': [10, 30, 13, 19, 25, 33, 23],
       'par_2': [1, 3, 1, 2, 3, 3, 2],
       'outcome': [0, 1, 1, 1, 1, 1 , 0]}

df = pd.DataFrame(dic)

variables = df.iloc[:,:-1]
results = df.iloc[:,-1]

multi_output_clf = MultiOutputClassifier(LogisticRegression(solver='lbfgs',
                                                            multi_class='ovr'))
multi_output_clf.fit(results.values.reshape(-1, 1),variables)

multi_output_clf.predict([[1]])
# array([[13,  3]])
```


Question:
How to get the first canonical correlation from sklearn's CCA module? In scikit-learn for Python, there is a module call cross_decomposition with a canonical correlation analysis (CCA) class. I have been trying to figure out how to give the class 2 multidimensional vectors of shape (n,m) and get the first canonical correlation coefficient. Looking off the documentation, a little example script is as follows.
```
from sklearn.cross_decomposition import CCA
import numpy as np

U = np.random.random_sample(500).reshape(100,5)
V = np.random.random_sample(500).reshape(100,5)

cca = CCA(n_components=1)
cca.fit(U, V)

cca.coef_.shape                   # (5,5)

U_c, V_c = cca.transform(U, V)

U_c.shape                         # (100,1)
V_c.shape                         # (100,1)
```
I do not really understand how to use this class to get the first canonical correlation between two matrices, which is all that I need. It seems generally directed towards classification and prediction problems, but I just need the first canonical correlation coefficient and nothing else. I know there are a few other posts somewhat similar to this, but the question remains unanswered and the best suggestion is to change to MATLAB, which is a non-solution. Any help is appreciated.

Response:
Given your transformed matrices U_c and V_c, you can indeed retrieve canonical component correlations like you did, and more generally for a CCA with n_comp CCs:
```py
result = np.corrcoef(U_c.T, V_c.T).diagonal(offset=n_comp)
```
Now, you do not have to tranform your data yourself, it has been done during the fitting procedure at least for the training data. The score are stored in the CCA instance by scikit-learn, so:
```py
score = np.diag(np.corrcoef(cca.x_scores_, cca.y_scores_, rowvar=False)[:n_comp, n_comp:])
```
Will give the same result, a vector of n_comp scalar values, corresponding to the score, or correlations between each pair of canonical components.


Question:
How to get classes labels from cross_val_predict used with predict_proba in scikit-learn I need to train a Random Forest classifier using a 3-fold cross-validation. For each sample, I need to retrieve the prediction probability when it happens to be in the test set. I am using scikit-learn version 0.18.dev0. This new version adds the feature to use the method cross_val_predict() with an additional parameter method to define which kind of prediction require from the estimator. In my case I want to use the predict_proba() method, which returns the probability for each class, in a multiclass scenario. However, when I run the method, I get as a result the matrix of prediction probabilities, where each rows represents a sample, and each column represents the prediction probability for a specific class. The problem is that the method does not indicate which class corresponds to each column. The value I need is the same (in my case using a RandomForestClassifier) returned in the attribute classes_ defined as: classes_ : array of shape = [n_classes] or a list of such arrays The classes labels (single output problem), or a list of arrays of class labels (multi-output problem). which is needed by predict_proba() because in its documentation it is written that: The order of the classes corresponds to that in the attribute classes_. A minimal example is the following:
```
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_predict

clf = RandomForestClassifier()

X = np.random.randn(10, 10)
y = y = np.array([1] * 4 + [0] * 3 + [2] * 3)

# how to get classes from here?
proba = cross_val_predict(estimator=clf, X=X, y=y, method="predict_proba")

# using the classifier without cross-validation
# it is possible to get the classes in this way:
clf.fit(X, y)
proba = clf.predict_proba(X)
classes = clf.classes_
```

Response: Yes, they will be in sorted order; this is because DecisionTreeClassifier (which is the default base_estimator for RandomForestClassifier) uses np.unique to construct the classes_ attribute which returns the sorted unique values of the input array.


Question:
Are predictions on scikit-learn models thread-safe? Given some classifier (SVC/Forest/NN/whatever) is it safe to call .predict on the same instance concurrently from different threads? From a distant point of view, my guess is they do not mutate any internal state. But I did not find anything in the docs about it. Here is a minimal example showing what I mean:
```py
#!/usr/bin/env python3
import threading

from sklearn import datasets
from sklearn import svm
from sklearn.ensemble import RandomForestClassifier
from sklearn.neural_network import MLPClassifier

X, y = datasets.load_iris(return_X_y=True)

# Some model. Might be any type, e.g.:
clf = svm.SVC()
clf = RandomForestClassifier(),
clf = MLPClassifier(solver='lbfgs')

clf.fit(X, y)

def use_model_for_predictions():
    for _ in range(10000):
        clf.predict(X[0:1])

# Is this safe?
thread_1 = threading.Thread(target=use_model_for_predictions)
thread_2 = threading.Thread(target=use_model_for_predictions)
thread_1.start()
thread_2.start()
```

Response:
Check out this Q&A, the predict and predict_proba methods should be thread safe as they only call NumPy, they do not affect model itself in any case so answer to your question is yes. You can find some info as well in replies here. For example in naive bayes the code is following:
```
def predict(self, X):
    """
    Perform classification on an array of test vectors X.
    Parameters
    ----------
    X : array-like of shape (n_samples, n_features)
    Returns
    -------
    C : ndarray of shape (n_samples,)
        Predicted target values for X
    """
    check_is_fitted(self)
    X = self._check_X(X)
    jll = self._joint_log_likelihood(X)
    return self.classes_[np.argmax(jll, axis=1)]
```
You can see that the first two lines are only checks for input. Abstract method _joint_log_likelihood is the one that interests us, described as:
```
@abstractmethod
def _joint_log_likelihood(self, X):
    """Compute the unnormalized posterior log probability of X
    I.e. ``log P(c) + log P(x|c)`` for all rows x of X, as an array-like of
    shape (n_classes, n_samples).
    Input is passed to _joint_log_likelihood as-is by predict,
    predict_proba and predict_log_proba.
    """
```
And finally for example for multinominal NB the function looks like (source):
```
def _joint_log_likelihood(self, X):
    """
    Compute the unnormalized posterior log probability of X, which is
    the features' joint log probability (feature log probability times
    the number of times that word appeared in that document) times the
    class prior (since we're working in log space, it becomes an addition)
    """
    joint_prob = X * self.feature_log_prob_.T + self.class_log_prior_
    return joint_prob
```
You can see that there is nothing thread unsafe in predict. Of course you can go through codes and check that for any of those classifiers :)


Question:
cluster points after KMeans clustering (scikit learn) I have done clustering using Kmeans using sklearn. While it has a method to print the centroids, I am finding it rather bizzare that scikit-learn doesn't have a method to print out the cluster-points of each cluster (or that I have not seen it so far). Is there a neat way to get the cluster-points of each cluster? I currently have this rather cludgy code to do it, where V is the dataset:
```
def getClusterPoints(V, labels):
    clusters = {}
    for l in range(0, max(labels)+1):
        data_points = []
        indices = [i for i, x in enumerate(labels) if x == l]
        for idx in indices:
            data_points.append(V[idx])
        clusters[l] = data_points
    return clusters
```
Suggestions/links are much appreciated. Thanks! PD.

Response:
For example
```
import numpy as np
from sklearn.cluster import KMeans
from sklearn import datasets

iris = datasets.load_iris()
X = iris.data
y = iris.target

estimator = KMeans(n_clusters=3)
estimator.fit(X)
```
You can get clusters of each point by
```
estimator.labels_
```
Out:
```
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
   0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
   0, 0, 0, 0, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
   1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
   1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 1, 1,
   2, 2, 2, 2, 1, 2, 1, 2, 1, 2, 2, 1, 1, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2,
   1, 2, 2, 2, 1, 2, 2, 2, 1, 2, 2, 1], dtype=int32)
```
Then get the indices of points for each cluster
```
{i: np.where(estimator.labels_ == i)[0] for i in range(estimator.n_clusters)}
```
Out:
```
{0: array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
        17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
        34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]),
 1: array([ 50,  51,  53,  54,  55,  56,  57,  58,  59,  60,  61,  62,  63,
         64,  65,  66,  67,  68,  69,  70,  71,  72,  73,  74,  75,  76,
         78,  79,  80,  81,  82,  83,  84,  85,  86,  87,  88,  89,  90,
         91,  92,  93,  94,  95,  96,  97,  98,  99, 101, 106, 113, 114,
        119, 121, 123, 126, 127, 133, 138, 142, 146, 149]),
 2: array([ 52,  77, 100, 102, 103, 104, 105, 107, 108, 109, 110, 111, 112,
        115, 116, 117, 118, 120, 122, 124, 125, 128, 129, 130, 131, 132,
        134, 135, 136, 137, 139, 140, 141, 143, 144, 145, 147, 148])}
```
Edit If you want to use array of points in X as values rather than the array of indices:
```
{i: X[np.where(estimator.labels_ == i)] for i in range(estimator.n_clusters)}
```


Question: HDBSCAN difference between parameters I'm confused about the difference between the following parameters in HDBSCAN min_cluster_size min_samples cluster_selection_epsilon Correct me if I'm wrong. For min_samples, if it is set to 7, then clusters formed need to have 7 or more points. For cluster_selection_epsilon if it is set to 0.5 meters, than any clusters that are more than 0.5 meters apart will not be merged into one. Meaning that each cluster will only include points that are 0.5 meters apart or less. How is that different from min_cluster_size?

Response: They technically do two different things. min_samples = the minimum number of neighbours to a core point. The higher this is, the more points are going to be discarded as noise/outliers. This is from DBScan part of HDBScan. min_cluster_size = the minimum size a final cluster can be. The higher this is, the bigger your clusters will be. This is from the H part of HDBScan. Increasing min_samples will increase the size of the clusters, but it does so by discarding data as outliers using DBSCAN. Increasing min_cluster_size while keeping min_samples small, by comparison, keeps those outliers but instead merges any smaller clusters with their most similar neighbour until all clusters are above min_cluster_size. So: If you want many highly specific clusters, use a small min_samples and a small min_cluster_size. If you want more generalized clusters but still want to keep most detail, use a small min_samples and a large min_cluster_size If you want very very general clusters and to discard a lot of noise in the clusters, use a large min_samples and a large min_cluster_size. (It's not possible to use min_samples larger than min_cluster_size, afaik)


Question:
How to print intercept and slope of a simple linear regression in Python with scikit-learn? I am trying to predict car prices (by machine learning) with a simple linear regression (only one independent variable). The variables are "highway miles per gallon"
```
0      27
1      27
2      26
3      30
4      22
       ..
200    28
201    25
202    23
203    27
204    25
Name: highway-mpg, Length: 205, dtype: int64
```
and "price":
```
0      13495.0
1      16500.0
2      16500.0
3      13950.0
4      17450.0
        ...
200    16845.0
201    19045.0
202    21485.0
203    22470.0
204    22625.0
Name: price, Length: 205, dtype: float64
```
With the following code:
```
from sklearn.linear_model import LinearRegression

x = df["highway-mpg"]
y = df["price"]
lm = LinearRegression()

lm.fit([x],[y])
Yhat = lm.predict([x])

print(Yhat)
print(lm.intercept_)
print(lm.coef_)
```
However, the intercept and slope coefficient print commands give me the following output:
```
[[0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 ...
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]]
```
Why doesn't it print the intercept and slope coefficient? The "Yhat" print command does print out the predicted values in an array properly, but somehow the other print commands do not print my desired output...

Response:
Essentially, what caused the strange looking coef_ and intercept_ was the fact that your data had 205 features and 205 targets with only 1 sample. Definitely not what you wanted! You probably want 1 feature, 205 samples, and 1 target. To do this, you need to reshape your data:
```
from sklearn.linear_model import LinearRegression
import numpy as np

mpg = np.array([27, 27, 26, 30, 22, 28, 25, 23, 27, 25]).reshape(-1, 1)
price = np.array([13495.0, 16500.0, 16500.0, 13950.0, 17450.0, 16845.0, 19045.0, 21485.0, 22470.0, 22625.0])

lm = LinearRegression()
lm.fit(mpg, price)

print(lm.intercept_)
print(lm.coef_)
```
I used the arrays there for testing, but obviously use the data from your dataframe. P.S. If you omit the resize, you get an error message like this:
```
ValueError: Expected 2D array, got 1D array instead:
array=[27 27 26 30 22 28 25 23 27 25].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
```
^ It tells you what to do!