Question: How to penalize False Negatives more than False Positives From the business perspective, false negatives lead to about tenfold higher costs (real money) than false positives. Given my standard binary classification models (logit, random forest, etc.), how can I incorporate this into my model? Do I have to change (weight) the loss function in favor of the 'preferred' error (FP) ? If so, how to do that?

Response:
There are several options for you: As suggested in the comments, class_weight should boost the loss function towards the preferred class. This option is supported by various estimators, including sklearn.linear_model.LogisticRegression, sklearn.svm.SVC, sklearn.ensemble.RandomForestClassifier, and others. Note there's no theoretical limit to the weight ratio, so even if 1 to 100 isn't strong enough for you, you can go on with 1 to 500, etc. You can also select the decision threshold very low during the cross-validation to pick the model that gives highest recall (though possibly low precision). The recall close to 1.0 effectively means false_negatives close to 0.0, which is what to want. For that, use sklearn.model_selection.cross_val_predict and sklearn.metrics.precision_recall_curve functions:
```
y_scores = cross_val_predict(classifier, x_train, y_train, cv=3,
                             method="decision_function")

precisions, recalls, thresholds = precision_recall_curve(y_train, y_scores)
```
If you plot the precisions and recalls against the thresholds, you should see the picture like this: After picking the best threshold, you can use the raw scores from classifier.decision_function() method for your final classification. Finally, try not to over-optimize your classifier, because you can easily end up with a trivial const classifier (which is obviously never wrong, but is useless).


Question:
ValueError: Data is not binary and pos_label is not specified I am trying to calculate roc_auc_score, but I am getting following error.
```
"ValueError: Data is not binary and pos_label is not specified"
```
My code snippet is as follows:
```
import numpy as np
from sklearn.metrics import roc_auc_score
y_scores=np.array([ 0.63, 0.53, 0.36, 0.02, 0.70 ,1 , 0.48, 0.46, 0.57])
y_true=np.array(['0', '1', '0', '0', '1', '1', '1', '1', '1'])
roc_auc_score(y_true, y_scores)
```
Please tell me what is wrong with it.

Response:
You only need to change y_trueso it looks like this:
```
y_true=np.array([0, 1, 0, 0, 1, 1, 1, 1, 1])
```
Explanation: If you take a look to what roc_auc_score functions does in https://github.com/scikit-learn/scikit-learn/blob/0.15.X/sklearn/metrics/metrics.py you will see that y_true is evaluated as follows:
```
classes = np.unique(y_true)
if (pos_label is None and not (np.all(classes == [0, 1]) or
 np.all(classes == [-1, 1]) or
 np.all(classes == [0]) or
 np.all(classes == [-1]) or
 np.all(classes == [1]))):
    raise ValueError("Data is not binary and pos_label is not specified")
```
At the moment of the execution pos_label is None, but as long as your are defining y_true as an array of characters the np.all are always false and as all of them are negated then the if condition is trueand the exception is raised.


Question:
How to compare ROC AUC scores of different binary classifiers and assess statistical significance in Python? (p-value, confidence interval) I would like to compare different binary classifiers in Python. For that, I want to calculate the ROC AUC scores, measure the 95% confidence interval (CI), and p-value to access statistical significance. Below is a minimal example in scikit-learn which trains three different models on a binary classification dataset, plots the ROC curves and calculates the AUC scores. Here are my specific questions: How to calculate the 95% confidence interval (CI) of the ROC AUC scores on the test set? (e.g. with bootstrapping). How to compare the AUC scores (on test set) and measure the p-value to assess statistical significance? (The null hypothesis is that the models are not different. Rejecting the null hypothesis means the difference in AUC scores is statistically significant.) .
```
import numpy as np

np.random.seed(2018)

from sklearn.datasets import load_breast_cancer
from sklearn.metrics import roc_auc_score, roc_curve
from sklearn.model_selection import train_test_split

from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import RandomForestClassifier
from sklearn.neural_network import MLPClassifier
import matplotlib
import matplotlib.pyplot as plt

data = load_breast_cancer()

X = data.data
y = data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=17)

# Naive Bayes Classifier
nb_clf = GaussianNB()
nb_clf.fit(X_train, y_train)
nb_prediction_proba = nb_clf.predict_proba(X_test)[:, 1]

# Ranodm Forest Classifier
rf_clf = RandomForestClassifier(n_estimators=20)
rf_clf.fit(X_train, y_train)
rf_prediction_proba = rf_clf.predict_proba(X_test)[:, 1]

# Multi-layer Perceptron Classifier
mlp_clf = MLPClassifier(alpha=1, hidden_layer_sizes=150)
mlp_clf.fit(X_train, y_train)
mlp_prediction_proba = mlp_clf.predict_proba(X_test)[:, 1]

def roc_curve_and_score(y_test, pred_proba):
    fpr, tpr, _ = roc_curve(y_test.ravel(), pred_proba.ravel())
    roc_auc = roc_auc_score(y_test.ravel(), pred_proba.ravel())
    return fpr, tpr, roc_auc

plt.figure(figsize=(8, 6))
matplotlib.rcParams.update({'font.size': 14})
plt.grid()
fpr, tpr, roc_auc = roc_curve_and_score(y_test, rf_prediction_proba)
plt.plot(fpr, tpr, color='darkorange', lw=2,
         label='ROC AUC={0:.3f}'.format(roc_auc))
fpr, tpr, roc_auc = roc_curve_and_score(y_test, nb_prediction_proba)
plt.plot(fpr, tpr, color='green', lw=2,
         label='ROC AUC={0:.3f}'.format(roc_auc))
fpr, tpr, roc_auc = roc_curve_and_score(y_test, mlp_prediction_proba)
plt.plot(fpr, tpr, color='crimson', lw=2,
         label='ROC AUC={0:.3f}'.format(roc_auc))
plt.plot([0, 1], [0, 1], color='navy', lw=1, linestyle='--')
plt.legend(loc="lower right")
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('1 - Specificity')
plt.ylabel('Sensitivity')
plt.show()
```

Response:
Bootstrap for 95% confidence interval You want to repeat your analysis on multiple resamplings of your data. In the general case, assume you have a function f(x) that determines whatever statistic you need from data x and you can bootstrap like this:
```
def bootstrap(x, f, nsamples=1000):
    stats = [f(x[np.random.randint(x.shape[0], size=x.shape[0])]) for _ in range(nsamples)]
    return np.percentile(stats, (2.5, 97.5))
```
This gives you so-called plug-in estimates of the 95% confidence interval (i.e. you just take the percentiles of the bootstrap distribution). In your case, you can write a more specific function like this
```
def bootstrap_auc(clf, X_train, y_train, X_test, y_test, nsamples=1000):
    auc_values = []
    for b in range(nsamples):
        idx = np.random.randint(X_train.shape[0], size=X_train.shape[0])
        clf.fit(X_train[idx], y_train[idx])
        pred = clf.predict_proba(X_test)[:, 1]
        roc_auc = roc_auc_score(y_test.ravel(), pred.ravel())
        auc_values.append(roc_auc)
    return np.percentile(auc_values, (2.5, 97.5))
```
Here, clf is the classifier for which you want to test the performance and X_train, y_train, X_test, y_test are like in your code. This gives me the following confidence intervals (rounded to three digits, 1000 bootstrap samples): Naive Bayes: 0.986 [0.980 0.988] (estimate, lower and upper limit of confidence interval) Random Forest: 0.983 [0.974 0.989] Multilayer Perceptron: 0.974 [0.223 0.98] Permutation tests to test against chance performance A permutation test would technically go over all permutations of your observation sequence and evaluate your roc curve with the permuted target values (features are not permuted). This is ok if you have a few observations, but it becomes very costly if you more observations. It is therefore common to subsample the number of permutations and simply do a number of random permutations. Here, the implementation depends a bit more on the specific thing you want to test. The following function does that for your roc_auc values
```
def permutation_test(clf, X_train, y_train, X_test, y_test, nsamples=1000):
    idx1 = np.arange(X_train.shape[0])
    idx2 = np.arange(X_test.shape[0])
    auc_values = np.empty(nsamples)
    for b in range(nsamples):
        np.random.shuffle(idx1)  # Shuffles in-place
        np.random.shuffle(idx2)
        clf.fit(X_train, y_train[idx1])
        pred = clf.predict_proba(X_test)[:, 1]
        roc_auc = roc_auc_score(y_test[idx2].ravel(), pred.ravel())
        auc_values[b] = roc_auc
    clf.fit(X_train, y_train)
    pred = clf.predict_proba(X_test)[:, 1]
    roc_auc = roc_auc_score(y_test.ravel(), pred.ravel())
    return roc_auc, np.mean(auc_values >= roc_auc)
```
This function again takes your classifier as clf and returns the AUC value on the unshuffled data and the p-value (i.e. probability to observe an AUC value larger than or equal to what you have in the unshuffled data). Running this with 1000 samples gives p-values of 0 for all three classifiers. Note that these are not exact because of the sampling, but they are an indicating that all of these classifiers perform better than chance. Permutation test for differences between classifiers This is much easier. Given two classifiers, you have prediction for every observation. You just shuffle the assignment between predictions and classifiers like this
```
def permutation_test_between_clfs(y_test, pred_proba_1, pred_proba_2, nsamples=1000):
    auc_differences = []
    auc1 = roc_auc_score(y_test.ravel(), pred_proba_1.ravel())
    auc2 = roc_auc_score(y_test.ravel(), pred_proba_2.ravel())
    observed_difference = auc1 - auc2
    for _ in range(nsamples):
        mask = np.random.randint(2, size=len(pred_proba_1.ravel()))
        p1 = np.where(mask, pred_proba_1.ravel(), pred_proba_2.ravel())
        p2 = np.where(mask, pred_proba_2.ravel(), pred_proba_1.ravel())
        auc1 = roc_auc_score(y_test.ravel(), p1)
        auc2 = roc_auc_score(y_test.ravel(), p2)
        auc_differences.append(auc1 - auc2)
    return observed_difference, np.mean(auc_differences >= observed_difference)
```
With this test and 1000 samples, I find no significant differences between the three classifiers: Naive bayes vs random forest: diff=0.0029, p(diff>)=0.311 Naive bayes vs MLP: diff=0.0117, p(diff>)=0.186 random forest vs MLP: diff=0.0088, p(diff>)=0.203 Where diff denotes the difference in roc curves between the two classifiers and p(diff>) is the empirical probability to observe a larger difference on a shuffled data set.


Question:
List of all classification algorithms [closed] Closed. This question needs to be more focused. It is not currently accepting answers. Want to improve this question? Guide the asker to update the question so it focuses on a single, specific problem. Narrowing the question will help others answer the question concisely. You may edit the question if you feel you can improve it yourself. If edited, the question will be reviewed and might be reopened. Closed 2 years ago. The community reviewed whether to reopen this question 2 years ago and left it closed: Original close reason(s) were not resolved Improve this question I have a classification problem and I would like to test all the available algorithms to test their performance in tackling the problem. If you know any classification algorithm other than these listed below, please list it here.
```
GradientBoostingClassifier()
DecisionTreeClassifier()
RandomForestClassifier()
LinearDiscriminantAnalysis()
LogisticRegression()
KNeighborsClassifier()
GaussianNB()
ExtraTreesClassifier()
BaggingClassifier()
```

Response:
The answers did not provide the full list of classifiers, so I have listed them below.
```
from sklearn.tree import ExtraTreeClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm.classes import OneClassSVM
from sklearn.neural_network.multilayer_perceptron import MLPClassifier
from sklearn.neighbors.classification import RadiusNeighborsClassifier
from sklearn.neighbors.classification import KNeighborsClassifier
from sklearn.multioutput import ClassifierChain
from sklearn.multioutput import MultiOutputClassifier
from sklearn.multiclass import OutputCodeClassifier
from sklearn.multiclass import OneVsOneClassifier
from sklearn.multiclass import OneVsRestClassifier
from sklearn.linear_model.stochastic_gradient import SGDClassifier
from sklearn.linear_model.ridge import RidgeClassifierCV
from sklearn.linear_model.ridge import RidgeClassifier
from sklearn.linear_model.passive_aggressive import PassiveAggressiveClassifier
from sklearn.gaussian_process.gpc import GaussianProcessClassifier
from sklearn.ensemble.voting_classifier import VotingClassifier
from sklearn.ensemble.weight_boosting import AdaBoostClassifier
from sklearn.ensemble.gradient_boosting import GradientBoostingClassifier
from sklearn.ensemble.bagging import BaggingClassifier
from sklearn.ensemble.forest import ExtraTreesClassifier
from sklearn.ensemble.forest import RandomForestClassifier
from sklearn.naive_bayes import BernoulliNB
from sklearn.calibration import CalibratedClassifierCV
from sklearn.naive_bayes import GaussianNB
from sklearn.semi_supervised import LabelPropagation
from sklearn.semi_supervised import LabelSpreading
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.svm import LinearSVC
from sklearn.linear_model import LogisticRegression
from sklearn.linear_model import LogisticRegressionCV
from sklearn.naive_bayes import MultinomialNB
from sklearn.neighbors import NearestCentroid
from sklearn.svm import NuSVC
from sklearn.linear_model import Perceptron
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
from sklearn.svm import SVC
from sklearn.mixture import DPGMM
from sklearn.mixture import GMM
from sklearn.mixture import GaussianMixture
from sklearn.mixture import VBGMM
```


Question: How do I calculate the Adjusted R-squared score using scikit-learn? I'm already using the r2_score function but don't understand how I can get the "adjusted" R^2 score from this. The description at this page doesn't mention it - maybe it's the adjusted score by default?

Response: Adjusted R2 requires number of independent variables as well. That's why it will not be calculated using such an independent metrics function (as we are not providing, how ypred was calculated). However you can calculate the adjusted R2 from R2 with a simple formula given here where n is number of observations in sample and p is number of independent variables in model


Question: python tsne.transform does not exist? I am trying to transform two datasets: x_train and x_test using tsne. I assume the way to do this is to fit tsne to x_train, and then transform x_test and x_train. But, I am not able to transform any of the datasets. tsne = TSNE(random_state = 420, n_components=2, verbose=1, perplexity=5, n_iter=350).fit(x_train) I assume that tsne has been fitted to x_train. But, when I do this: x_train_tse = tsne.transform(x_subset) I get: AttributeError: 'TSNE' object has no attribute 'transform' Any help will be appreciated. (I know I could do fit_transform, but wouldn't I get the same error on x_test?)

Response: Judging by the documentation of sklearn, TSNE simply does not have any transform method. Also, TSNE is an unsupervised method for dimesionality reduction/visualization, so it does not really work with a TRAIN and TEST. You simply take all of your data and use fit_transform to have the transformation and plot it. EDIT - It is actually not possible to learn a transformation and reuse it on different data (i.e. Train and Test), as T-sne does not learn a mapping function on a lower dimensional space, but rather runs an iterative procedure on a subspace to find an equilibrium that minimizes a loss/distance ON SOME DATA. Therefore if you want to preprocess and reduce dimensionality of both a Train and Test datasets, the way to go is PCA/SVD or Autoencoders. T-Sne will only help you for unsupervised tasks :)


Question: How do I create a sklearn.datasets.base.Bunch object in scikit-learn from my own data? In most of the Scikit-learn algorithms, the data must be loaded as a Bunch object. For many example in the tutorial load_files() or other functions are used to populate the Bunch object. Functions like load_files() expect data to be present in certain format, but I have data stored in a different format, namely a CSV file with strings for each field. How do I parse this and load data in the Bunch object format?

Response:
You can do it like this:
```
import numpy as np
import sklearn.datasets

examples = []
examples.append('some text')
examples.append('another example text')
examples.append('example 3')

target = np.zeros((3,), dtype=np.int64)
target[0] = 0
target[1] = 1
target[2] = 0
dataset = sklearn.datasets.base.Bunch(data=examples, target=target)
```


Question: how to split a dataset into training and validation set keeping ratio between classes? I have a multi class classification problem and my dataset is skewed, I have 100 instances of a particular class and say 10 of some different class, so I want to split my dataset keeping ratio between classes, if I have 100 instances of a particular class and I want 30% of records to go in the training set I want to have there 30 instances of my 100 record represented class and 3 instances of my 10 record represented class and so on.

Response:
You can use sklearn's StratifiedKFold, from the online docs: Stratified K-Folds cross validation iterator Provides train/test indices to split data in train test sets. This cross-validation object is a variation of KFold that returns stratified folds. The folds are made by preserving the percentage of samples for each class.
```

```python
from sklearn import cross_validation
```

```python
X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])
```

```python
y = np.array([0, 0, 1, 1])
```

```python
skf = cross_validation.StratifiedKFold(y, n_folds=2)
```

```python
len(skf)
#Output
#2
```

```python
print(skf)
#Output
#sklearn.cross_validation.StratifiedKFold(labels=[0 0 1 1], n_folds=2,
#                                         shuffle=False, random_state=None)
```

```python
for train_index, test_index in skf:
   print("TRAIN:", train_index, "TEST:", test_index)
   X_train, X_test = X[train_index], X[test_index]
   y_train, y_test = y[train_index], y[test_index]
#Output
#TRAIN: [1 3] TEST: [0 2]
#TRAIN: [0 2] TEST: [1 3]
#```
#This will preserve your class ratios so that the splits retain the class ratios, this will work fine with pandas dfs. As suggested by @Ali_m you could use StratifiedShuffledSplit which accepts a split ratio param: sss = StratifiedShuffleSplit(y, 3, test_size=0.7, random_state=0) would produce a 70% split.
```


Question:
PLS-DA algorithm in python Partial Least Squares (PLS) algorithm is implemented in the scikit-learn library, as documented here: http://scikit-learn.org/0.12/auto_examples/plot_pls.html In the case where y is a binary vector, a variant of this algorithm is being used, the Partial least squares Discriminant Analysis (PLS-DA) algorithm. Does the PLSRegression module in sklearn.pls implements also this binary case? If not, where can I find a python implementation for it? In my binary case, I'm trying to use the PLSRegression:
```
pls = PLSRegression(n_components=10)
pls.fit(x, y)
x_r, y_r = pls.transform(x, y, copy=True)
```
In the transform function, the code gets exception in this line:
```
y_scores = np.dot(Yc, self.y_rotations_)
```
The error message is "ValueError: matrices are not aligned". Yc is the normalized y vector, and self.y_rotations_ = [1.]. In the fit function, self.y_rotations_ = np.ones(1) if the original y is a univariate vector (y.shape1=1).

Response:
PLS-DA is really a "trick" to use PLS for categorical outcomes instead of the usual continuous vector/matrix. The trick consists of creating a dummy identity matrix of zeros/ones which represents membership to each of the categories. So if you have a binary outcome to be predicted (i.e. male/female , yes/no, etc) your dummy matrix will have TWO columns representing the membership to either category. For example, consider the outcome gender for four people: 2 males and 2 females. The dummy matrix should be coded as :
```
import numpy as np
dummy=np.array([[1,1,0,0],[0,0,1,1]]).T
```
, where each column represents the membership to the two categories (male, female) Then your model for data in variable Xdata ( shape 4 rows,arbitrary columns ) would be:
```
myplsda=PLSRegression().fit(X=Xdata,Y=dummy)
```
The predicted categories can be extracted from comparison of the two indicator variables in mypred:
```
mypred= myplsda.predict(Xdata)
```
For each row/case the predicted gender is that with the highest predicted membership.


Question:
Get selected feature names TFIDF Vectorizer I'm using python and I want to get the TFIDF representation for a large corpus of data, I'm using the following code to convert the docs into their TFIDF form.
```
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf_vectorizer = TfidfVectorizer(
    min_df=1,  # min count for relevant vocabulary
    max_features=4000,  # maximum number of features
    strip_accents='unicode',  # replace all accented unicode char
    # by their corresponding  ASCII char
    analyzer='word',  # features made of words
    token_pattern=r'\w{1,}',  # tokenize only words of 4+ chars
    ngram_range=(1, 1),  # features made of a single tokens
    use_idf=True,  # enable inverse-document-frequency reweighting
    smooth_idf=True,  # prevents zero division for unseen words
    sublinear_tf=False)

tfidf_df = tfidf_vectorizer.fit_transform(df['text'])
```
Here I pass a parameter max_features. The vectorizer will select the best features and return a scipy sparse matrix. Problem is I dont know which features are getting selected and how do I map those feature names back to the scipy matrix I get? Basically for the n selected features from the m number of documents, I want a m x n matrix with the selected features as the column names instead of their integer ids. How do I accomplish this?

Response: You can use tfidf_vectorizer.get_feature_names(). This will print feature names selected (terms selected) from the raw documents. You can also use tfidf_vectorizer.vocabulary_ attribute to get a dict which will map the feature names to their indices, but will not be sorted. The array from get_feature_names() will be sorted by index.


Question:
How to add k-means predicted clusters in a column to a dataframe in Python I have a question about kmeans clustering in python. So I did the analysis that way:
```
from sklearn.cluster import KMeans

km = KMeans(n_clusters=12, random_state=1)
new = data._get_numeric_data().dropna(axis=1)
km.fit(new)
predict=km.predict(new)
```
How can I add the column with cluster results to my first dataframe "data" as an additional column? Thanks!

Response:
Assuming the column length is as the same as each column in you dataframe df, all you need to do is this:
```
df['NEW_COLUMN'] = pd.Series(predict, index=df.index)
```


Question: How to show loss values during training in scikit-learn? I want check my loss values during the training time so I can observe the loss at each iteration. So far I haven't found an easy way for scikit learn to give me a history of loss values, nor did I find a functionality already within scikit to plot the loss for me. If there was no way to plot this, it'd be great if I could simply fetch the final loss values at the end of classifier.fit. Note: I am aware of the fact that some solutions are closed form. I'm using several classifiers which do not have analytical solutions, such as logistic regression and svm. Does anyone have any suggestions?

Response:
So I couldn't find very good documentation on directly fetching the loss values per iteration, but I hope this will help someone in the future:
```
old_stdout = sys.stdout
sys.stdout = mystdout = StringIO()
clf = SGDClassifier(**kwargs, verbose=1)
clf.fit(X_tr, y_tr)
sys.stdout = old_stdout
loss_history = mystdout.getvalue()
loss_list = []
for line in loss_history.split('\n'):
    if(len(line.split("loss: ")) == 1):
        continue
    loss_list.append(float(line.split("loss: ")[-1]))
plt.figure()
plt.plot(np.arange(len(loss_list)), loss_list)
plt.savefig("warmstart_plots/pure_SGD:"+str(kwargs)+".png")
plt.xlabel("Time in epochs")
plt.ylabel("Loss")
plt.close()
```
This code will take a normal SGDClassifier(just about any linear classifier), and intercept the verbose=1 flag, and will then split to get the loss from the verbose printing. Obviously this is slower but will give us the loss and print it.


Question: How to tune GaussianNB? Trying to fit data with GaussianNB() gives me low accuracy score. I'd like to try Grid Search, but it seems that parameters sigma and theta cannot be set. Is there anyway to tune GausssianNB?

Response:
You can tune 'var_smoothing' parameter like this:
```
nb_classifier = GaussianNB()

params_NB = {'var_smoothing': np.logspace(0,-9, num=100)}
gs_NB = GridSearchCV(estimator=nb_classifier,
                 param_grid=params_NB,
                 cv=cv_method,   # use any cross validation technique
                 verbose=1,
                 scoring='accuracy')
gs_NB.fit(x_train, y_train)

gs_NB.best_params_
```


Question:
Gaussian Mixture Models: Difference between Spark MLlib and scikit-learn I'm trying to use Gaussian Mixture models on a sample of a dataset. I used bothMLlib (with pyspark) and scikit-learn and get very different results, the scikit-learn one looking more realistic.
```
from pyspark.mllib.clustering import GaussianMixture as SparkGaussianMixture
from sklearn.mixture import GaussianMixture
from pyspark.mllib.linalg import Vectors
```
Scikit-learn:
```
local = pd.DataFrame([ x.asDict() for x in df.sample(0.0001).collect() ])
model1 = GaussianMixture(n_components=3)
model1.fit([ [x] for x in local['field'].tolist() ])

model1.means_
array([[7.56123598e+00],
   [1.32517410e+07],
   [3.96762639e+04]])

model1.covariances_
array([[[6.65177423e+00]],
   [[1.00000000e-06]],
   [[8.38380897e+10]]])
```
MLLib:
```
model2 = SparkGaussianMixture.train(
    sc.createDataFrame(local).rdd.map(lambda x: Vectors.dense(x.field)),
    k=3,
    convergenceTol=1e-4,
    maxIterations=100
)

model2.gaussians
[MultivariateGaussian(mu=DenseVector([28736.5113]), sigma=DenseMatrix(1, 1, [1094083795.0001], 0)),
 MultivariateGaussian(mu=DenseVector([7839059.9208]), sigma=DenseMatrix(1, 1, [38775218707109.83], 0)),
 MultivariateGaussian(mu=DenseVector([43.8723]), sigma=DenseMatrix(1, 1, [608204.4711], 0))]
```
However, I'm interested in running the entire dataset through the model which I'm afraid would require parallelizing (and hence use MLlib) to get results in finite time. Am I doing anything wrong / missing something? Data: The complete data has an extremely long tail and looks like: whereas the data has a clearly normal dist ceneterd somewhere closer to one clustered by scikit-learn: I am using Spark 2.3.0 (AWS EMR). Edit: Initialization params:
```
local = pd.DataFrame([ x.asDict() for x in df.sample(0.0001).collect() ])
model1 = GaussianMixture(n_components=3, init_params='random')
model1.fit([ [x] for x in local['field'].tolist() ])

model1.means_
array([[2.17611913e+04],
   [8.03184505e+06],
   [7.56871801e+00]])

model1.covariances_
rray([[[1.01835902e+09]],
   [[3.98552130e+13]],
   [[6.95161493e+00]]])
```

Response: This isn't a python problem, per se. It seems to be more of a machine learning/data validation/data segmentation question, IMO. That being said, you are correct in thinking that you must parallelize your work, but it matters in what ways you do it. There are things like 8-bit quantization and model parallelism in your model that you may look into to help you get at what you're after: training a model on large datasets, in a timely manner, without sacrificing data quality or fidelity. Here is a blog post about quantization: https://petewarden.com/2016/05/03/how-to-quantize-neural-networks-with-tensorflow/ Here is a blog post about model parallelism and 8-bit quantization from Tim Dettmers' Blog: http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/ and the associated paper: https://arxiv.org/pdf/1511.04561.pdf Though you will want to keep in mind that, depending on FP operations on your GPU, you may not see substantial benefit from this route: https://blog.inten.to/hardware-for-deep-learning-part-3-gpu-8906c1644664 HTH and YMMV. Also, you may want to look into data folding, but can't remember the details nor the paper I read at this point in time. I'll land this here to remember once I do though.


Question: AdaBoostClassifier with different base learners I am trying to use AdaBoostClassifier with a base learner other than DecisionTree. I have tried SVM and KNeighborsClassifier but I get errors. What are the classifiers that can be used with AdaBoostClassifier?

Response:
Ok, we have a systematic method to find out all the base learners supported by AdaBoostClassifier. Compatible base learner's fit method needs to support sample_weight, which can be obtained by running following code:
```
import inspect
from sklearn.utils.testing import all_estimators
for name, clf in all_estimators(type_filter='classifier'):
    if 'sample_weight' in inspect.getargspec(clf().fit)[0]:
       print name
```
This results in following output:
```
AdaBoostClassifier,
BernoulliNB,
DecisionTreeClassifier,
ExtraTreeClassifier,
ExtraTreesClassifier,
MultinomialNB,
NuSVC,
Perceptron,
RandomForestClassifier,
RidgeClassifierCV,
SGDClassifier,
SVC.
```
If the classifier doesn't implement predict_proba, you will have to set AdaBoostClassifier parameter algorithm = 'SAMME'.


Question: Python sklearn - how to calculate p-values This is probably a simple question but I am trying to calculate the p-values for my features either using classifiers for a classification problem or regressors for regression. Could someone suggest what is the best method for each case and provide sample code? I want to just see the p-value for each feature rather than keep the k best / percentile of features etc as explained in the documentation. Thank you

Response:
You can use statsmodels
```py
import statsmodels.api as sm
logit_model=sm.Logit(y_train,X_train)
result=logit_model.fit()
print(result.summary())
```
The results would be something like this
```
Logit Regression Results
==============================================================================
Dep. Variable:                      y   No. Observations:               406723
Model:                          Logit   Df Residuals:                   406710
Method:                           MLE   Df Model:                           12
Date:                Fri, 12 Apr 2019   Pseudo R-squ.:                0.001661
Time:                        16:48:45   Log-Likelihood:            -2.8145e+05
converged:                      False   LL-Null:                   -2.8192e+05
                                        LLR p-value:                8.758e-193
==============================================================================
                 coef    std err          z      P>|z|      [0.025      0.975]
------------------------------------------------------------------------------
x1            -0.0037      0.003     -1.078      0.281      -0.010       0.003
```


Question: How to use scipy.optimize.minimize function when you want to compute gradient along with the objective function? scipy.optimize.minimze takes obj and jac functions as input. and I believe it will call them separately as and when needed. But more often than not we come across objective functions whose gradient computation shares a lot of computations from the objective function. So ideally I would like to compute the obj and grad simultaneously. But this doesn't seem to be the case with this library? What is the way to deal with it if one still wants to use scipy.optimize.minimze if at all there is?

Response:
You totally can. Just use jac=True:
```

```python
import numpy as np
```

```python
from scipy.optimize import minimize
```

```python
def f_and_grad(x):
   ...:     return x**2, 2*x
   ...:
```

```python
minimize(f_and_grad, [1], jac=True)
#Output
#fun: 1.8367099231598242e-40
# hess_inv: array([[ 0.5]])
#      jac: array([  2.71050543e-20])
#  message: 'Optimization terminated successfully.'
#     nfev: 4
#      nit: 2
#     njev: 4
#   status: 0
#  success: True
#        x: array([  1.35525272e-20])
#```
#It's actually documented: jac : bool or callable, optional Jacobian (gradient) of objective function. Only for CG, BFGS, Newton-CG, L-BFGS-B, TNC, SLSQP, dogleg, trust-ncg. If jac is a Boolean and is True, fun is assumed to return the gradient along with the objective function. If False, the gradient will be estimated numerically. jac can also be a callable returning the gradient of the objective. In this case, it must accept the same arguments as fun. (emphasis mine)
```


Question: Computing TF-IDF on the whole dataset or only on training data? In the chapter seven of this book "TensorFlow Machine Learning Cookbook" the author in pre-processing data uses fit_transform function of scikit-learn to get the tfidf features of text for training. The author gives all text data to the function before separating it into train and test. Is it a true action or we must separate data first and then perform fit_transform on train and transform on test?

Response: According to the documentation of scikit-learn, fit() is used in order to Learn vocabulary and idf from training set. On the other hand, fit_transform() is used in order to Learn vocabulary and idf, return term-document matrix. while transform() Transforms documents to document-term matrix. On the training set you need to apply both fit() and transform() (or just fit_transform() that essentially joins both operations) however, on the testing set you only need to transform() the testing instances (i.e. the documents). Remember that training sets are used for learning purposes (learning is achieved through fit()) while testing set is used in order to evaluate whether the trained model can generalise well to new unseen data points. For more details you can refer to the article fit() vs transform() vs fit_transform()


Question:
Convert sklearn.svm SVC classifier to Keras implementation I'm trying to convert some old code from using sklearn to Keras implementation. Since it is crucial to maintain the same way of operation, I want to understand if I'm doing it correctly. I've converted most of the code already, however I'm having trouble with sklearn.svm SVC classifier conversion. Here is how it looks right now:
```
from sklearn.svm import SVC
model = SVC(kernel='linear', probability=True)
model.fit(X, Y_labels)
```
Super easy, right. However, I couldn't find the analog of SVC classifier in Keras. So, what I've tried is this:
```
from keras.models import Sequential
from keras.layers import Dense

model = Sequential()
model.add(Dense(64, activation='relu'))
model.add(Dense(1, activation='softmax'))
model.compile(loss='squared_hinge',
              optimizer='adadelta',
              metrics=['accuracy'])
model.fit(X, Y_labels)
```
But, I think that it is not correct by any means. Could you, please, help me find an alternative of the SVC classifier from sklearn in Keras? Thank you.

Response:
If you are making a classifier, you need squared_hinge and regularizer, to get the complete SVM loss function as can be seen here. So you will also need to break your last layer to add regularization parameter before performing activation, I have added the code here. These changes should give you the output
```
from keras.regularizers import l2
from keras.models import Sequential
from keras.layers import Dense

model = Sequential()
model.add(Dense(64, activation='relu'))
model.add(Dense(1), kernel_regularizer=l2(0.01))
model.add(activation('softmax'))
model.compile(loss='squared_hinge',
              optimizer='adadelta',
              metrics=['accuracy'])
model.fit(X, Y_labels)
```
Also hinge is implemented in keras for binary classification, so if you are working on a binary classification model, use the code below.
```
from keras.regularizers import l2
from keras.models import Sequential
from keras.layers import Dense

model = Sequential()
model.add(Dense(64, activation='relu'))
model.add(Dense(1), kernel_regularizer=l2(0.01))
model.add(activation('linear'))
model.compile(loss='hinge',
              optimizer='adadelta',
              metrics=['accuracy'])
model.fit(X, Y_labels)
```
If you cannot understand the article or have issues with the code, feel free to comment. I had this same issue a while back, and this GitHub thread helped me understand, maybe go through it too, some of the ideas here are directly from here https://github.com/keras-team/keras/issues/2588


Question: How to set a threshold for a sklearn classifier based on ROC results? I trained an ExtraTreesClassifier (gini index) using scikit-learn and it suits my needs fairly. Not so good accuracy, but using a 10-fold cross validation, AUC is 0.95. I would like to use this classifier on my work. I am quite new to ML, so please forgive me if I'm asking you something conceptually wrong. I plotted some ROC curves, and by it, its seems I have a specific threshold where my classifier starts performing well. I'd like to set this value on the fitted classifier, so everytime I'd call predict, the classifiers use that threshold and I could believe in the FP and TP rates. I also came to this post (scikit .predict() default threshold), where its stated that a threshold is not a generic concept for classifiers. But since the ExtraTreesClassifier has the method predict_proba, and the ROC curve is also related to thresdholds definition, it seems to me I should be available to specify it. I did not find any parameter, nor any class/interface to use to do it. How can I set a threshold for it for a trained ExtraTreesClassifier (or any other one) using scikit-learn? Many Thanks, Colis

Response:
This is what I have done:
```
model = SomeSklearnModel()
model.fit(X_train, y_train)
predict = model.predict(X_test)
predict_probabilities = model.predict_proba(X_test)
fpr, tpr, _ = roc_curve(y_test, predict_probabilities)
```
However, I am annoyed that predict chooses a threshold corresponding to 0.4% of true positives (false positives are zero). The ROC curve shows a threshold I like better for my problem where the true positives are approximately 20% (false positive around 4%). I then scan the predict_probabilities to find what probability value corresponds to my favourite ROC point. In my case this probability is 0.21. Then I create my own predict array:
```
predict_mine = np.where(rf_predict_probabilities > 0.21, 1, 0)
```
and there you go:
```
confusion_matrix(y_test, predict_mine)
```
returns what I wanted:
```
array([[6927,  309],
       [ 621,  121]])
```


Question:
TypeError: ufunc 'true_divide' output (typecode 'd') could not be coerced to provided output parameter (typecode 'q') I am trying to apply Gower distance implementation to my data frame. While it was smoothly working with the same dataset with more features, this time it gives an error when I call the Gower distance function. I import the Gower's function from another .py code in the same directory. Here is my code:
```py
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

import gower_function as gf

# Importing the dataset with pandas
dataset = pd.read_excel('input_partial.xlsx')
X = dataset.iloc[:, 1:].values
df = pd.DataFrame(X)
#obtaining gower distances of instances
Gower = gf.gower_distances(X)
```
and after executing this, I got the error below:
```none
File "<ipython-input-10-6a4c39600b0e>", line 1, in <module>
Gower = gf.gower_distances(X)

File "C:\Users\...\Clustering\Section 24 - K-Means
Clustering\gower_function.py", line 184, in gower_distances
X_num = np.divide(X_num ,max_of_numeric,out=np.zeros_like(X_num),
where=max_of_numeric!=0)

TypeError: ufunc 'true_divide' output (typecode 'd') could not be coerced to
provided output parameter (typecode 'q') according to the casting rule
''same_kind''
```
I did not understand how it can give this error on the same dataset with only fewer features (columns). Is there anyone who can recognize the reason?

Response:
You need to specify dtype of the left operand to be non integer, some floating point type, e.g.
```
a = np.array(..., dtype=float)
np.divide(a, b, out=np.zeros_like(a), where=(b!=0))
```
If dtype of a and b are both integers, then the error you get is: No loop matching the specified signature and casting was found for ufunc true_divide. If dtype of a is integer but b - float, then the error you get is: ufunc 'true_divide' output (typecode 'd') could not be coerced to provided output parameter (typecode 'l') according to the casting rule ''same_kind''.


Question:
Grid Search and Early Stopping Using Cross Validation with XGBoost in SciKit-Learn I am fairly new to sci-kit learn and have been trying to hyper-paramater tune XGBoost. My aim is to use early stopping and grid search to tune the model parameters and use early stopping to control the number of trees and avoid overfitting. As I am using cross validation for the grid search, I was hoping to also use cross-validation in the early stopping criteria. The code I have so far looks like this:
```
import numpy as np
import pandas as pd
from sklearn import model_selection
import xgboost as xgb

#Import training and test data
train = pd.read_csv("train.csv").fillna(value=-999.0)
test = pd.read_csv("test.csv").fillna(value=-999.0)

# Encode variables
y_train = train.price_doc
x_train = train.drop(["id", "timestamp", "price_doc"], axis=1)

# XGBoost - sklearn method
gbm = xgb.XGBRegressor()

xgb_params = {
'learning_rate': [0.01, 0.1],
'n_estimators': [2000],
'max_depth': [3, 5, 7, 9],
'gamma': [0, 1],
'subsample': [0.7, 1],
'colsample_bytree': [0.7, 1]
}

fit_params = {
'early_stopping_rounds': 30,
'eval_metric': 'mae',
'eval_set': [[x_train,y_train]]
}

grid = model_selection.GridSearchCV(gbm, xgb_params, cv=5,
fit_params=fit_params)
grid.fit(x_train,y_train)
```
The problem I am having is the 'eval_set' parameter. I understand that this wants the predictor and response variables but I am not sure how I can use the cross-validation data as the early stopping criteria. Does anyone know how to overcome this problem? Thanks.

Response:
You could pass you early_stopping_rounds, and eval_set as an extra fit_params to GridSearchCV, and that would actually work. However, GridSearchCV will not change the fit_params between the different folds, so you would end up using the same eval_set in all the folds, which might not be what you mean by CV.
```
model=xgb.XGBClassifier()
clf = GridSearchCV(model, parameters,
                         fit_params={'early_stopping_rounds':20,\
                         'eval_set':[(X,y)]},cv=kfold)
```
After some tweaking, I found the safest way to integrate early_stopping_rounds and the sklearn API is to implement an early_stopping mechanism your self. You can do it if you do a GridSearchCV with n_rounds as paramter to be tuned. You can then watch the mean_validation_score for the different models with increasing n_rounds. Then you can define a custom heuristic for early stop. it wont save the computational time needed to evaluate all the possible n_rounds though I think it is also a better approach then using a single split hold-out for this purpose.


Question:
How to normalize the Train and Test data using MinMaxScaler sklearn So, I have this doubt and have been looking for answers. So the question is when I use,
```
from sklearn import preprocessing
min_max_scaler = preprocessing.MinMaxScaler()

df = pd.DataFrame({'A':[1,2,3,7,9,15,16,1,5,6,2,4,8,9],'B':[15,12,10,11,8,14,17,20,4,12,4,5,17,19],'C':['Y','Y','Y','Y','N','N','N','Y','N','Y','N','N','Y','Y']})

df[['A','B']] = min_max_scaler.fit_transform(df[['A','B']])
df['C'] = df['C'].apply(lambda x: 0 if x.strip()=='N' else 1)
```
After which I will train and test the model (A,B as features, C as Label) and get some accuracy score. Now my doubt is, what happens when I have to predict the label for new set of data. Say,
```
df = pd.DataFrame({'A':[25,67,24,76,23],'B':[2,54,22,75,19]})
```
Because when I normalize the column the values of A and B will be changed according to the new data, not the data which the model will be trained on. So, now my data after the data preparation step that is as below, will be.
```
data[['A','B']] = min_max_scaler.fit_transform(data[['A','B']])
```
Values of A and B will change with respect to the Max and Min value of df[['A','B']]. The data prep of df[['A','B']] is with respect to Min Max of df[['A','B']]. How can the data preparation be valid with respect to different numbers relate? I don't understand how the prediction will be correct here.

Response:
You should fit the MinMaxScaler using the training data and then apply the scaler on the testing data before the prediction. In summary: Step 1: fit the scaler on the TRAINING data Step 2: use the scaler to transform the TRAINING data Step 3: use the transformed training data to fit the predictive model Step 4: use the scaler to transform the TEST data Step 5: predict using the trained model (step 3) and the transformed TEST data (step 4). Example using your data:
```
from sklearn import preprocessing
min_max_scaler = preprocessing.MinMaxScaler()
#training data
df = pd.DataFrame({'A':[1,2,3,7,9,15,16,1,5,6,2,4,8,9],'B':[15,12,10,11,8,14,17,20,4,12,4,5,17,19],'C':['Y','Y','Y','Y','N','N','N','Y','N','Y','N','N','Y','Y']})
#fit and transform the training data and use them for the model training
df[['A','B']] = min_max_scaler.fit_transform(df[['A','B']])
df['C'] = df['C'].apply(lambda x: 0 if x.strip()=='N' else 1)

#fit the model
model.fit(df['A','B'])

#after the model training on the transformed training data define the testing data df_test
df_test = pd.DataFrame({'A':[25,67,24,76,23],'B':[2,54,22,75,19]})

#before the prediction of the test data, ONLY APPLY the scaler on them
df_test[['A','B']] = min_max_scaler.transform(df_test[['A','B']])

#test the model
y_predicted_from_model = model.predict(df_test['A','B'])
```
Example using iris data:
```
import matplotlib.pyplot as plt
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from sklearn.svm import SVC

data = datasets.load_iris()
X = data.data
y = data.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_train)

model = SVC()
model.fit(X_train_scaled, y_train)

X_test_scaled = scaler.transform(X_test)
y_pred = model.predict(X_test_scaled)
```
Hope this helps. See also by post here: https://towardsdatascience.com/everything-you-need-to-know-about-min-max-normalization-in-python-b79592732b79


Question:
CountVectorizer: AttributeError: 'numpy.ndarray' object has no attribute 'lower' I have a one-dimensional array with large strings in each of the elements. I am trying to use a CountVectorizer to convert text data into numerical vectors. However, I am getting an error saying:
```
AttributeError: 'numpy.ndarray' object has no attribute 'lower'
```
mealarray contains large strings in each of the elements. There are 5000 such samples. I am trying to vectorize this as given below:
```
vectorizer = CountVectorizer(
    stop_words='english',
    ngram_range=(1, 1),  #ngram_range=(1, 1) is the default
    dtype='double',
)
data = vectorizer.fit_transform(mealarray)
```
The full stacktrace :
```
File "/Library/Python/2.7/site-packages/sklearn/feature_extraction/text.py", line 817, in fit_transform
    self.fixed_vocabulary_)
  File "/Library/Python/2.7/site-packages/sklearn/feature_extraction/text.py", line 748, in _count_vocab
    for feature in analyze(doc):
  File "/Library/Python/2.7/site-packages/sklearn/feature_extraction/text.py", line 234, in <lambda>
    tokenize(preprocess(self.decode(doc))), stop_words)
  File "/Library/Python/2.7/site-packages/sklearn/feature_extraction/text.py", line 200, in <lambda>
    return lambda x: strip_accents(x.lower())
AttributeError: 'numpy.ndarray' object has no attribute 'lower'
```

Response:
Check the shape of mealarray. If the argument to fit_transform is an array of strings, it must be a one-dimensional array. (That is, mealarray.shape must be of the form (n,).) For example, you'll get the "no attribute" error if mealarray has a shape such as (n, 1). You could try something like
```
data = vectorizer.fit_transform(mealarray.ravel())
```


Question: Python Non negative Matrix Factorization that handles both zeros and missing data? I look for a NMF implementation that has a python interface, and handles both missing data and zeros. I don't want to impute my missing values before starting the factorization, I want them to be ignored in the minimized function. It seems that neither scikit-learn, nor nimfa, nor graphlab, nor mahout propose such an option. Thanks!

Response:
Scipy has a method to solve non-negative least squares problem (NNLS). In this answer, I am reproducing my blogpost on using scipy's NNLS for non-negative matrix factorisation. You may also be interested in my other blog posts that use autograd, Tensorflow and CVXPY for NNMF. Goal: Our goal is given a matrix A, decompose it into two non-negative factors, as follows:
```
A (M×N) ≈ W (M×K) × H (K×N), such that  W (M×K) ≥ 0 and  H (K×N) ≥ 0
```
Overview Our solution consists of two steps. First, we fix W and learn H, given A. Next, we fix H and learn W, given A. We repeat this procedure iteratively. Fixing one variable and learning the other (in this setting) is popularly known as alternating least squares, as the problem is reduced to a least squares problem. However, an important thing to note is that since we want to constraint W and H to be non-negative, we us NNLS instead of least squares. Step 1: Learning H, given A and W Using the illustration above, we can learn each column of H, using the corresponding column from A and the matrix W.
```
H[:,j]=NNLS(W,A[:,j])
```
Handling missing entries in A In the problem of collaborative filtering, A is usually the user-item matrix and it has a lot of missing entries. These missing entries correspond to user who have not rated items. We can modify our formulation to account for these missing entries. Consider that M' ≤ M entries in A have observed data, we would now modify the above equation as:
```
H[:,j]=NNLS(W[mask],A[:,j][mask])
```
where, the mask is found by considering only the M′ entries. Step 2: Learning W, given A and H Code example Imports
```
import numpy as np
import pandas as pd
```
Creating matrix to be factorised
```
M, N = 20, 10

np.random.seed(0)
A_orig = np.abs(np.random.uniform(low=0.0, high=1.0, size=(M,N)))
print pd.DataFrame(A_orig).head()

0   1   2   3   4   5   6   7   8   9
0   0.548814    0.715189    0.602763    0.544883    0.423655    0.645894    0.437587    0.891773    0.963663    0.383442
1   0.791725    0.528895    0.568045    0.925597    0.071036    0.087129    0.020218    0.832620    0.778157    0.870012
2   0.978618    0.799159    0.461479    0.780529    0.118274    0.639921    0.143353    0.944669    0.521848    0.414662
3   0.264556    0.774234    0.456150    0.568434    0.018790    0.617635    0.612096    0.616934    0.943748    0.681820
4   0.359508    0.437032    0.697631    0.060225    0.666767    0.670638    0.210383    0.128926    0.315428    0.363711
```
Masking a few entries
```
A = A_orig.copy()
A[0, 0] = np.NAN
A[3, 1] = np.NAN
A[6, 3] = np.NAN

A_df = pd.DataFrame(A)
print A_df.head()

0   1   2   3   4   5   6   7   8   9
0   NaN 0.715189    0.602763    0.544883    0.423655    0.645894    0.437587    0.891773    0.963663    0.383442
1   0.791725    0.528895    0.568045    0.925597    0.071036    0.087129    0.020218    0.832620    0.778157    0.870012
2   0.978618    0.799159    0.461479    0.780529    0.118274    0.639921    0.143353    0.944669    0.521848    0.414662
3   0.264556    NaN 0.456150    0.568434    0.018790    0.617635    0.612096    0.616934    0.943748    0.681820
4   0.359508    0.437032    0.697631    0.060225    0.666767    0.670638    0.210383    0.128926    0.315428    0.363711
```
Defining matrices W and H
```
K = 4
W = np.abs(np.random.uniform(low=0, high=1, size=(M, K)))
H = np.abs(np.random.uniform(low=0, high=1, size=(K, N)))
W = np.divide(W, K*W.max())
H = np.divide(H, K*H.max())

pd.DataFrame(W).head()

0   1   2   3
0   0.078709    0.175784    0.095359    0.045339
1   0.006230    0.016976    0.171505    0.114531
2   0.135453    0.226355    0.250000    0.054753
3   0.167387    0.066473    0.005213    0.191444
4   0.080785    0.096801    0.148514    0.209789

pd.DataFrame(H).head()

0   1   2   3   4   5   6   7   8   9
0   0.074611    0.216164    0.157328    0.003370    0.088415    0.037721    0.250000    0.121806    0.126649    0.162827
1   0.093851    0.034858    0.209333    0.048340    0.130195    0.057117    0.024914    0.219537    0.247731    0.244654
2   0.230833    0.197093    0.084828    0.020651    0.103694    0.059133    0.033735    0.013604    0.184756    0.002910
3   0.196210    0.037417    0.020248    0.022815    0.171121    0.062477    0.107081    0.141921    0.219119    0.185125
```
Defining the cost that we want to minimise
```
def cost(A, W, H):
    from numpy import linalg
    WH = np.dot(W, H)
    A_WH = A-WH
    return linalg.norm(A_WH, 'fro')
```
However, since A has missing entries, we have to define the cost in terms of the entries present in A
```
def cost(A, W, H):
    from numpy import linalg
    mask = pd.DataFrame(A).notnull().values
    WH = np.dot(W, H)
    WH_mask = WH[mask]
    A_mask = A[mask]
    A_WH_mask = A_mask-WH_mask
    # Since now A_WH_mask is a vector, we use L2 instead of Frobenius norm for matrix
    return linalg.norm(A_WH_mask, 2)
```
Let us just try to see the cost of the initial set of values of W and H we randomly assigned.
```
cost(A, W, H)

7.3719938519859509
```
Alternating NNLS procedure
```
num_iter = 1000
num_display_cost = max(int(num_iter/10), 1)
from scipy.optimize import nnls

for i in range(num_iter):
    if i%2 ==0:
        # Learn H, given A and W
        for j in range(N):
            mask_rows = pd.Series(A[:,j]).notnull()
            H[:,j] = nnls(W[mask_rows], A[:,j][mask_rows])[0]
    else:
        for j in range(M):
            mask_rows = pd.Series(A[j,:]).notnull()
            W[j,:] = nnls(H.transpose()[mask_rows], A[j,:][mask_rows])[0]
    WH = np.dot(W, H)
    c = cost(A, W, H)
    if i%num_display_cost==0:
        print i, c

0 4.03939072472
100 2.38059096458
200 2.35814781954
300 2.35717011529
400 2.35711130357
500 2.3571079918
600 2.35710729854
700 2.35710713129
800 2.35710709085
900 2.35710708109

A_pred = pd.DataFrame(np.dot(W, H))
A_pred.head()

0   1   2   3   4   5   6   7   8   9
0   0.564235    0.677712    0.558999    0.631337    0.536069    0.621925    0.629133    0.656010    0.839802    0.545072
1   0.788734    0.539729    0.517534    1.041272    0.119894    0.448402    0.172808    0.658696    0.493093    0.825311
2   0.749886    0.575154    0.558981    0.931156    0.270149    0.502035    0.287008    0.656178    0.588916    0.741519
3   0.377419    0.743081    0.370408    0.637094    0.071684    0.529433    0.767696    0.628507    0.832910    0.605742
4   0.458661    0.327143    0.610012    0.233134    0.685559    0.377750    0.281483    0.269960    0.468756    0.114950
```
Let's view the values of the masked entries.
```
A_pred.values[~pd.DataFrame(A).notnull().values]

array([ 0.56423481,  0.74308143,  0.10283106])

Original values were:

A_orig[~pd.DataFrame(A).notnull().values]

array([ 0.5488135 ,  0.77423369,  0.13818295])
```


Question:
How to customize sklearn cross validation iterator by indices? Similar to Custom cross validation split sklearn I want to define my own splits for GridSearchCV for which I need to customize the built in cross-validation iterator. I want to pass my own set of train-test indices for cross validation to the GridSearch instead of allowing the iterator to determine them for me. I went through the available cv iterators on the sklearn documentation page but couldn't find it. For example I want to implement something like this Data has 9 samples For 2 fold cv I create my own set of training-testing indices
```

```python
train_indices = [[1,3,5,7,9],[2,4,6,8]]
```

```python
test_indices = [[2,4,6,8],[1,3,5,7,9]]
#Output
#                 1st fold^    2nd fold^
```

```python
custom_cv = sklearn.cross_validation.customcv(train_indices,test_indices)
```

```python
clf = GridSearchCV(X,y,params,cv=custom_cv)
#Output
#```
#What can be used to work like customcv?
```

Response:
Actually, cross-validation iterators are just that: Iterators. They give back a tuple of train/test fold at each iteration. This should then work for you:
```
custom_cv = zip(train_indices, test_indices)
```
Also, for the specific case you are mentioning, you can do
```
import numpy as np
labels = np.arange(0, 10) % 2
from sklearn.cross_validation import LeaveOneLabelOut
cv = LeaveOneLabelOut(labels)
```
Observe that list(cv) yields
```
[(array([1, 3, 5, 7, 9]), array([0, 2, 4, 6, 8])),
 (array([0, 2, 4, 6, 8]), array([1, 3, 5, 7, 9]))]
```


Question:
TF*IDF for Search Queries Okay, so I have been following these two posts on TF*IDF but am little confused : http://css.dzone.com/articles/machine-learning-text-feature Basically, I want to create a search query that contains searches through multiple documents. I would like to use the scikit-learn toolkit as well as the NLTK library for Python The problem is that I don't see where the two TF*IDF vectors come from. I need one search query and multiple documents to search. I figured that I calculate the TF*IDF scores of each document against each query and find the cosine similarity between them, and then rank them by sorting the scores in descending order. However, the code doesn't seem to come up with the right vectors. Whenever I reduce the query to only one search, it is returning a huge list of 0's which is really strange. Here is the code:
```
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from nltk.corpus import stopwords

train_set = ("The sky is blue.", "The sun is bright.") #Documents
test_set = ("The sun in the sky is bright.") #Query
stopWords = stopwords.words('english')

vectorizer = CountVectorizer(stop_words = stopWords)
transformer = TfidfTransformer()

trainVectorizerArray = vectorizer.fit_transform(train_set).toarray()
testVectorizerArray = vectorizer.transform(test_set).toarray()
print 'Fit Vectorizer to train set', trainVectorizerArray
print 'Transform Vectorizer to test set', testVectorizerArray

transformer.fit(trainVectorizerArray)
print transformer.transform(trainVectorizerArray).toarray()

transformer.fit(testVectorizerArray)

tfidf = transformer.transform(testVectorizerArray)
print tfidf.todense()
```

Response:
You're defining train_set and test_set as tuples, but I think that they should be lists:
```
train_set = ["The sky is blue.", "The sun is bright."] #Documents
test_set = ["The sun in the sky is bright."] #Query
```
Using this the code seems to run fine.


Question:
how to use GridSearchCV with custom estimator in sklearn? I have an estimator that should be compatible with the sklearn api. I am trying to fit one parameter of this estimator with gridsearchcv but I do not understand how to do it. This is my code:
```
import numpy as np
import sklearn as sk

from sklearn.linear_model import LinearRegression, LassoLarsCV, RidgeCV
from sklearn.linear_model.base import LinearClassifierMixin, SparseCoefMixin, BaseEstimator

class ELM(BaseEstimator):

    def __init__(self, n_nodes, link='rbf', output_function='lasso', n_jobs=1, c=1):
        self.n_jobs = n_jobs
        self.n_nodes = n_nodes
        self.c = c

        if link == 'rbf':
            self.link = lambda z: np.exp(-z*z)
        elif link == 'sig':
            self.link = lambda z: 1./(1 + np.exp(-z))
        elif link == 'id':
            self.link = lambda z: z
        else:
            self.link = link

        if output_function == 'lasso':
            self.output_function = LassoLarsCV(cv=10, n_jobs=self.n_jobs)
        elif output_function == 'lr':
            self.output_function = LinearRegression(n_jobs=self.n_jobs)

        elif output_function == 'ridge':
            self.output_function = RidgeCV(cv=10)

        else:
            self.output_function = output_function

        return

    def H(self, x):

        n, p = x.shape
        xw = np.dot(x, self.w.T)
        xw = xw + np.ones((n, 1)).dot(self.b.T)
        return self.link(xw)

    def fit(self, x, y, w=None):

        n, p = x.shape
        self.mean_y = y.mean()
        if w == None:
            self.w = np.random.uniform(-self.c, self.c, (self.n_nodes, p))
        else:
            self.w = w

        self.b = np.random.uniform(-self.c, self.c, (self.n_nodes, 1))
        self.h_train = self.H(x)
        self.output_function.fit(self.h_train, y)

        return self

    def predict(self, x):
        self.h_predict = self.H(x)
        return self.output_function.predict(self.h_predict)

    def get_params(self, deep=True):
        return {"n_nodes": self.n_nodes,
                "link": self.link,
                "output_function": self.output_function,
                "n_jobs": self.n_jobs,
                "c": self.c}

    def set_params(self, **parameters):
        for parameter, value in parameters.items():
            setattr(self, parameter, value)

### Fit the c parameter ###
X = np.random.normal(0, 1, (100,5))
y = X[:,1] * X[:,2] + np.random.normal(0, .1, 100)

gs = sk.grid_search.GridSearchCV(ELM(n_nodes=20, output_function='lr'),
                                 cv=5,
                                 param_grid={"c":np.linspace(0.0001,1,10)},
                                 fit_params={})

#gs.fit(X, y) # Error
```

Response: There are 2 problems within your code: You didn't specify scoring argument to GridSearchCV. You seem to be doing regression, so mean_squared_error is an option. Your set_params doesn't return reference to the object itself. You should add return self after the for loop. As Andreas already mentioned, you rarely need to redefine set_params and get_params in scikit-learn. Just having inherited from the BaseEstimator should be enough.


Question: sklearn metrics.log_loss is positive vs. scoring 'neg_log_loss' is negative Making sure I am getting this right: If we use sklearn.metrics.log_loss standalone, i.e. log_loss(y_true,y_pred), it generates a positive score -- the smaller the score, the better the performance. However, if we use 'neg_log_loss' as a scoring scheme as in 'cross_val_score", the score is negative -- the bigger the score, the better the performance. And this is due to the scoring scheme being built to be consistent with other scoring schemes - since, generally, the higher the better, we negate usual log_loss to be consistent with the trend. And it is done so solely for that purpose. Is this understanding correct? [Background: got positive scores for metric.log_loss, and negative scores for 'neg_los_loss', and both refer to the same documentation page.]

Response: The sklearn.metrics.log_loss is an implementation of the error metric as typically defined, and which is as most error metrics a positive number. In this case, it is a metric which is generally minimized (e.g. as mean squared error for regression), in contrast to metrics such as accuracy which is maximized. The neg_log_loss is hence a technicality to create a utility value, which allows optimizing functions and classes of sklearn to maximize this utility without having to change the function's behavior for each metric (such include for instance named cross_val_score, GridSearchCV, RandomizedSearchCV, and others).


Question:
How to find out the accuracy? I've wondered if there is a function in sklearn which corresponds to the accuracy(difference between actual and predicted data) and how to print it out?
```
from sklearn import datasets
iris = datasets.load_iris()
from sklearn.naive_bayes import GaussianNB
naive_classifier= GaussianNB()
y =naive_classifier.fit(iris.data, iris.target).predict(iris.data)
pr=naive_classifier.predict(iris.data)
```

Response:
Most classifiers in scikit have an inbuilt score() function, in which you can input your X_test and y_test and it will output the appropriate metric for that estimator. For classification estimators it is mostly 'mean accuracy'. Also sklearn.metrics have many functions available which will output different metrics like accuracy, precision, recall etc. For your specific question you need accuracy_score
```
from sklearn.metrics import accuracy_score
score = accuracy_score(iris.target, pr)
```


Question: Retrieve list of training features names from classifier Is there a way to retrieve the list of feature names used for training of a classifier, once it has been trained with the fit method? I would like to get this information before applying to unseen data. The data used for training is a pandas DataFrame and in my case, the classifier is a RandomForestClassifier.

Response:
I have a solution which works but is not very elegant. This is an old post with no existing solutions so I suppose there are not any. Create and fit your model. For example
```
model = GradientBoostingRegressor(**params)
model.fit(X_train, y_train)
```
Then you can add an attribute which is the 'feature_names' since you know them at training time
```
model.feature_names = list(X_train.columns.values)
```
I typically then put the model into a binary file to pass it around but you can ignore this
```
joblib.dump(model, filename)
loaded_model = joblib.load(filename)
```
Then you can get the feature names back from the model to use them when you predict
```
f_names = loaded_model.feature_names
loaded_model.predict(X_pred[f_names])
```


Question:
Python List of Ngrams with frequencies I need to get most popular ngrams from text. Ngrams length must be from 1 to 5 words. I know how to get bigrams and trigrams. For example:
```
bigram_measures = nltk.collocations.BigramAssocMeasures()
finder = nltk.collocations.BigramCollocationFinder.from_words(words)
finder.apply_freq_filter(3)
finder.apply_word_filter(filter_stops)
matches1 = finder.nbest(bigram_measures.pmi, 20)
```
However, i found out that scikit-learn can get ngrams with various length. For example I can get ngrams with length from 1 to 5.
```
v = CountVectorizer(analyzer=WordNGramAnalyzer(min_n=1, max_n=5))
```
But WordNGramAnalyzer is now deprecated. My question is: How can i get N best word collocations from my text, with collocations length from 1 to 5. Also i need to get FreqList of this collocations/ngrams. Can i do that with nltk/scikit ? I need to get combinations of ngrams with various lengths from one text ? For example using NLTK bigrams and trigrams where many situations in which my trigrams include my bitgrams, or my trigrams are part of bigger 4-grams. For example: bitgrams: hello my trigrams: hello my name I know how to exclude bigrams from trigrams, but i need better solutions.

Response:
update Since scikit-learn 0.14 the format has changed to:
```
n_grams = CountVectorizer(ngram_range=(1, 5))
```
Full example:
```
test_str1 = "I need to get most popular ngrams from text. Ngrams length must be from 1 to 5 words."
test_str2 = "I know how to exclude bigrams from trigrams, but i need better solutions."

from sklearn.feature_extraction.text import CountVectorizer

c_vec = CountVectorizer(ngram_range=(1, 5))

# input to fit_transform() should be an iterable with strings
ngrams = c_vec.fit_transform([test_str1, test_str2])

# needs to happen after fit_transform()
vocab = c_vec.vocabulary_

count_values = ngrams.toarray().sum(axis=0)

# output n-grams
for ng_count, ng_text in sorted([(count_values[i],k) for k,i in vocab.items()], reverse=True):
    print(ng_count, ng_text)
```
which outputs the following (note that the word I is removed not because it's a stopword (it's not) but because of its length: https://stackoverflow.com/a/20743758/):
```
> (3, u'to')
> (3, u'from')
> (2, u'ngrams')
> (2, u'need')
> (1, u'words')
> (1, u'trigrams but need better solutions')
> (1, u'trigrams but need better')
...
```
This should/could be much simpler these days, imo. You can try things like textacy, but that can come with its own complications sometimes, like initializing a Doc, which doesn't work currently with v.0.6.2 as shown on their docs. If doc initialization worked as promised, in theory the following would work (but it doesn't):
```
test_str1 = "I need to get most popular ngrams from text. Ngrams length must be from 1 to 5 words."
test_str2 = "I know how to exclude bigrams from trigrams, but i need better solutions."

import textacy

# some version of the following line
doc = textacy.Doc([test_str1, test_str2])

ngrams = doc.to_bag_of_terms(ngrams={1, 5}, as_strings=True)
print(ngrams)
```
old answer WordNGramAnalyzer is indeed deprecated since scikit-learn 0.11. Creating n-grams and getting term frequencies is now combined in sklearn.feature_extraction.text.CountVectorizer. You can create all n-grams ranging from 1 till 5 as follows:
```
n_grams = CountVectorizer(min_n=1, max_n=5)
```
More examples and information can be found in scikit-learn's documentation about text feature extraction.


Question: Python's Xgoost: ValueError('feature_names may not contain [, ] or <') Python's implementation of XGBClassifier does not accept the characters [, ] or <' as features names. If that occurs, it raises the following: ValueError('feature_names may not contain [, ] or <') It would seem that the obvious solution would be to pass the equivalent numpy arrays, and get rid of the column names altogether, but if they haven't done it that must be for a reason. What use does XGBoost have for the feature names, and what is the downside of simply passing it Numpy Arrays instead of Pandas DataFrames? Edit: this is not a question about workarounds (those are obvious and in the question), but about why it is implemented this way

Response:
I know it's late but writing this answer here for other folks who might face this. Here is what I found after facing this issue: This error typically happens if your column names have the symbols [ or ] or <. Here is an example:
```
import pandas as pd
import numpy as np
from xgboost.sklearn import XGBRegressor

# test input data with string, int, and symbol-included columns
df = pd.DataFrame({'0': np.random.randint(0, 2, size=100),
                   '[test1]': np.random.uniform(0, 1, size=100),
                   'test2': np.random.uniform(0, 1, size=100),
                  3: np.random.uniform(0, 1, size=100)})

target = df.iloc[:, 0]
predictors = df.iloc[:, 1:]

# basic xgb model
xgb0 = XGBRegressor(objective= 'reg:linear')
xgb0.fit(predictors, target)
```
The code above will throw an error:
```
ValueError: feature_names may not contain [, ] or <
```
But if you remove those square brackets from '[test1]' then it works fine. Below is a generic way of removing [, ] or < from your column names:
```
import re
import pandas as pd
import numpy as np
from xgboost.sklearn import XGBRegressor
regex = re.compile(r"\[|\]|<", re.IGNORECASE)

# test input data with string, int, and symbol-included columns
df = pd.DataFrame({'0': np.random.randint(0, 2, size=100),
                   '[test1]': np.random.uniform(0, 1, size=100),
                   'test2': np.random.uniform(0, 1, size=100),
                  3: np.random.uniform(0, 1, size=100)})

df.columns = [regex.sub("_", col) if any(x in str(col) for x in set(('[', ']', '<'))) else col for col in df.columns.values]

target = df.iloc[:, 0]
predictors = df.iloc[:, 1:]

# basic xgb model
xgb0 = XGBRegressor(objective= 'reg:linear')
xgb0.fit(predictors, target)
```
For more read this code line form xgboost core.py: xgboost/core.py. That's the check failing which the error is thrown.


Question: Does KMeans normalize features automatically in sklearn I was wondering if KMeans automatically normalizes the features before doing clustering. There seems to be no option to provide an input to ask for normalization.

Response: One differentiates data preprocessing (normalization, binning, weighting etc) and machine learning algorithms application. Use sklearn.preprocessing for data preprocessing. Moreover, data can be preprocessed in chain by different preprocessors. As for K-means, often it is not sufficient to normalize only mean. One normalizes data equalizing variance along different features as K-means is sensitive to variance in data, and features with larger variance have more emphasis on result. So for K-means, I would recommend using StandardScaler for data preprocessing. Don't forget also that k-means results are sensitive to the order of observations, and it is worth to run algorithm several times, shuffling data in between, averaging resulting clusters and running final evaluations with those averaged clusters centers as starting points.


Question:
DBSCAN with custom metric I have the following given: a dataset in the range of thousands a way of computing the similarity, but the datapoints themselves I cannot plot them in euclidian space I know that DBSCAN should support custom distance metric but I dont know how to use it. say I have a function
```
def similarity(x,y):
    return  similarity ...
```
and I have a list of data that can be passed pairwise into that function, how do I specify this when using the DBSCAN implementation of scikit-learn ? Ideally what I want to do is to get a list of the clusters but I cant figure out how to get started in the first place. There is a lot of terminology that still confuses me: http://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html How do I pass a feature array and what is it ? How do I fit this implementation to my needs ? How will I be able to get my "sublists" from this algorithm ?

Response:
A "feature array" is simply an array of the features of a datapoint in your dataset. metric is the parameter you're looking for. It can be a string (the name of a builtin metric), or a callable. Your similarity function is a callable. This isn't well described in the documentation, but a metric has to do just that, take two datapoints as parameters, and return a number.
```
def similarity(x, y):
    return ...

reduced_dataset = sklearn.cluster.DBSCAN(metric=similarity).fit(dataset)
```


Question:
LabelEncoder order of fit for a Pandas df I am fitting a scikit-learn LabelEncoder on a column in a pandas df. How is the order, in which the encountered strings are mapped to the integers, determined? Is it deterministic? More importantly, can I specify this order?
```
import pandas as pd
from sklearn import preprocessing

df = pd.DataFrame(data=["first", "second", "third", "fourth"], columns=['x'])
le = preprocessing.LabelEncoder()
le.fit(df['x'])
print list(le.classes_)
### this prints ['first', 'fourth', 'second', 'third']
encoded = le.transform(["first", "second", "third", "fourth"])
print encoded
### this prints [0 2 3 1]
```
I would expect le.classes_ to be ["first", "second", "third", "fourth"] and then encoded to be [0 1 2 3], since this is the order in which the strings appear in the column. Can this be done?

Response: It's done in sort order. In the case of strings, it is done in alphabetic order. There's no documentation for this, but looking at the source code for LabelEncoder.transform we can see the work is mostly delegated to the function numpy.setdiff1d, with the following documentation: Find the set difference of two arrays. Return the sorted, unique values in ar1 that are not in ar2. (Emphasis mine). Note that since this is not documented, it is probably implementation defined and can be changed between versions. It could be that just the version I looked use the sort order, and other versions of scikit-learn may change this behavior (by not using numpy.setdiff1d).


Question:
Use attribute and target matrices for TensorFlow Linear Regression Python I'm trying to follow this tutorial. TensorFlow just came out and I'm really trying to understand it. I'm familiar with penalized linear regression like Lasso, Ridge, and ElasticNet and its usage in scikit-learn. For scikit-learn Lasso regression, all I need to input into the regression algorithm is DF_X [an M x N dimensional attribute matrix (pd.DataFrame)] and SR_y [an M dimensional target vector (pd.Series)]. The Variable structure in TensorFlow is a bit new to me and I'm not sure how to structure my input data into what it wants. It seems as if softmax regression is for classification. How can I restructure my DF_X (M x N attribute matrix) and SR_y (M dimensional target vector) to input into tensorflow for linear regression? My current method for doing a Linear Regression uses pandas, numpy, and sklearn and it's shown below. I think this question will be really helpful for people getting familiar with TensorFlow:
```
#!/usr/bin/python
import pandas as pd
import numpy as np
import tensorflow as tf
from sklearn.linear_model import LassoCV

#Create DataFrames for attribute and target matrices
DF_X = pd.DataFrame(np.array([[0,0,1],[2,3,1],[4,5,1],[3,4,1]]),columns=["att1","att2","att3"],index=["s1","s2","s3","s4"])
SR_y = pd.Series(np.array([3,2,5,8]),index=["s1","s2","s3","s4"],name="target")

print DF_X
#att1  att2  att3
#s1     0     0     1
#s2     2     3     1
#s3     4     5     1
#s4     3     4     1

print SR_y
#s1    3
#s2    2
#s3    5
#s4    8
#Name: target, dtype: int64

#Create Linear Model (Lasso Regression)
model = LassoCV()
model.fit(DF_X,SR_y)

print model
#LassoCV(alphas=None, copy_X=True, cv=None, eps=0.001, fit_intercept=True,
#max_iter=1000, n_alphas=100, n_jobs=1, normalize=False, positive=False,
#precompute='auto', random_state=None, selection='cyclic', tol=0.0001,
#verbose=False)

print model.coef_
#[ 0.         0.3833346  0.       ]
```

Response:
Softmax is an only addition function (in logistic regression for example), it is not a model like
```
model = LassoCV()
model.fit(DF_X,SR_y)
```
Therefore you can't simply give it data with fit method. However, you can simply create your model with the help of TensorFlow functions. First of all, you have to create a computational graph, for example for linear regression you will create tensors with the size of your data. They are only tensors and you will give them your array in another part of the program.
```
import tensorflow as tf
x = tf.placeholder("float", [4, 3])
y_ = tf.placeholder("float",[4])
```
When you create two variables, that will contain initial weights of our model
```
W = tf.Variable(tf.zeros([3,1]))
b = tf.Variable(tf.zeros([1]))
```
And now you can create the model (you want to create regression, not classification therefore you don't need to use tf.nn.softmax )
```
y=tf.matmul(x,W) + b
```
As you have regression and linear model you will use
```
loss=tf.reduce_sum(tf.square(y_ - y))
```
Then we will train our model with the same step as in the tutorial
```
train_step = tf.train.GradientDescentOptimizer(0.01).minimize(loss)
```
Now that you created the computational graph you have to write one more part of the program, where you will use this graph to work with your data.
```
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
sess.run(train_step, feed_dict={x:np.asarray(DF_X),y_:np.asarray(SR_y)})
```
Here you give your data to this computational graph with the help of feed_dict. In TensorFlow you provide information in numpy arrays. If you want to see your mistake you can write
```
sess.run(loss,feed_dict={x:np.asarray(DF_X),y_:np.asarray(SR_y)})
```


Question:
Problems obtaining most informative features with scikit learn? Im triying to obtain the most informative features from a textual corpus. From this well answered question I know that this task could be done as follows:
```
def most_informative_feature_for_class(vectorizer, classifier, classlabel, n=10):
    labelid = list(classifier.classes_).index(classlabel)
    feature_names = vectorizer.get_feature_names()
    topn = sorted(zip(classifier.coef_[labelid], feature_names))[-n:]

    for coef, feat in topn:
        print classlabel, feat, coef
```
Then:
```
most_informative_feature_for_class(tfidf_vect, clf, 5)
```
For this classfier:
```
X = tfidf_vect.fit_transform(df['content'].values)
y = df['label'].values

from sklearn import cross_validation
X_train, X_test, y_train, y_test = cross_validation.train_test_split(X,
                                                    y, test_size=0.33)
clf = SVC(kernel='linear', C=1)
clf.fit(X, y)
prediction = clf.predict(X_test)
```
The problem is the output of most_informative_feature_for_class:
```
5 a_base_de_bien bastante   (0, 2451)   -0.210683496368
  (0, 3533) -0.173621065386
  (0, 8034) -0.135543062425
  (0, 10346)    -0.173621065386
  (0, 15231)    -0.154148294738
  (0, 18261)    -0.158890483047
  (0, 21083)    -0.297476572586
  (0, 434)  -0.0596263855375
  (0, 446)  -0.0753492277856
  (0, 769)  -0.0753492277856
  (0, 1118) -0.0753492277856
  (0, 1439) -0.0753492277856
  (0, 1605) -0.0753492277856
  (0, 1755) -0.0637950312345
  (0, 3504) -0.0753492277856
  (0, 3511) -0.115802483001
  (0, 4382) -0.0668983049212
  (0, 5247) -0.315713152154
  (0, 5396) -0.0753492277856
  (0, 5753) -0.0716096348446
  (0, 6507) -0.130661516772
  (0, 7978) -0.0753492277856
  (0, 8296) -0.144739048504
  (0, 8740) -0.0753492277856
  (0, 8906) -0.0753492277856
  : :
  (0, 23282)    0.418623443832
  (0, 4100) 0.385906085143
  (0, 15735)    0.207958503155
  (0, 16620)    0.385906085143
  (0, 19974)    0.0936828782325
  (0, 20304)    0.385906085143
  (0, 21721)    0.385906085143
  (0, 22308)    0.301270427482
  (0, 14903)    0.314164150621
  (0, 16904)    0.0653764031957
  (0, 20805)    0.0597723455204
  (0, 21878)    0.403750815828
  (0, 22582)    0.0226150073272
  (0, 6532) 0.525138162099
  (0, 6670) 0.525138162099
  (0, 10341)    0.525138162099
  (0, 13627)    0.278332617058
  (0, 1600) 0.326774799211
  (0, 2074) 0.310556919237
  (0, 5262) 0.176400451433
  (0, 6373) 0.290124806858
  (0, 8593) 0.290124806858
  (0, 12002)    0.282832270298
  (0, 15008)    0.290124806858
  (0, 19207)    0.326774799211
```
It is not returning the label nor the words. Why this is happening and how can I print the words and the labels?. Do you guys this is happening since I am using pandas to read the data?. Another thing I tried is the following, form this question:
```
def print_top10(vectorizer, clf, class_labels):
    """Prints features with the highest coefficient values, per class"""
    feature_names = vectorizer.get_feature_names()
    for i, class_label in enumerate(class_labels):
        top10 = np.argsort(clf.coef_[i])[-10:]
        print("%s: %s" % (class_label,
              " ".join(feature_names[j] for j in top10)))

print_top10(tfidf_vect,clf,y)
```
But I get this traceback: Traceback (most recent call last):
```
File "/Users/user/PycharmProjects/TESIS_FINAL/Classification/Supervised_learning/Final/experimentos/RBF/SVM_con_rbf.py", line 237, in <module>
    print_top10(tfidf_vect,clf,5)
  File "/Users/user/PycharmProjects/TESIS_FINAL/Classification/Supervised_learning/Final/experimentos/RBF/SVM_con_rbf.py", line 231, in print_top10
    for i, class_label in enumerate(class_labels):
TypeError: 'int' object is not iterable
```
Any idea of how to solve this, in order to get the features with the highest coefficient values?.

Response:
To solve this specifically for linear SVM, we first have to understand the formulation of the SVM in sklearn and the differences that it has to MultinomialNB. The reason why the most_informative_feature_for_class works for MultinomialNB is because the output of the coef_ is essentially the log probability of features given a class (and hence would be of size [nclass, n_features], due to the formulation of the naive bayes problem. But if we check the documentation for SVM, the coef_ is not that simple. Instead coef_ for (linear) SVM is [n_classes * (n_classes -1)/2, n_features] because each of the binary models are fitted to every possible class. If we do possess some knowledge on which particular coefficient we're interested in, we could alter the function to look like the following:
```
def most_informative_feature_for_class_svm(vectorizer, classifier,  classlabel, n=10):
    labelid = ?? # this is the coef we're interested in.
    feature_names = vectorizer.get_feature_names()
    svm_coef = classifier.coef_.toarray()
    topn = sorted(zip(svm_coef[labelid], feature_names))[-n:]

    for coef, feat in topn:
        print feat, coef
```
This would work as intended and print out the labels and the top n features according to the coefficient vector that you're after. As for getting the correct output for a particular class, that would depend on the assumptions and what you aim to output. I suggest reading through the multi-class documentation within the SVM documentation to get a feel for what you're after. So using the train.txt file which was described in this question, we can get some kind of output, though in this situation it isn't particularly descriptive or helpful to interpret. Hopefully this helps you.
```
import codecs, re, time
from itertools import chain

import numpy as np

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

trainfile = 'train.txt'

# Vectorizing data.
train = []
word_vectorizer = CountVectorizer(analyzer='word')
trainset = word_vectorizer.fit_transform(codecs.open(trainfile,'r','utf8'))
tags = ['bs','pt','es','sr']

# Training NB
mnb = MultinomialNB()
mnb.fit(trainset, tags)

from sklearn.svm import SVC
svcc = SVC(kernel='linear', C=1)
svcc.fit(trainset, tags)

def most_informative_feature_for_class(vectorizer, classifier, classlabel, n=10):
    labelid = list(classifier.classes_).index(classlabel)
    feature_names = vectorizer.get_feature_names()
    topn = sorted(zip(classifier.coef_[labelid], feature_names))[-n:]

    for coef, feat in topn:
        print classlabel, feat, coef

def most_informative_feature_for_class_svm(vectorizer, classifier,  n=10):
    labelid = 3 # this is the coef we're interested in.
    feature_names = vectorizer.get_feature_names()
    svm_coef = classifier.coef_.toarray()
    topn = sorted(zip(svm_coef[labelid], feature_names))[-n:]

    for coef, feat in topn:
        print feat, coef

most_informative_feature_for_class(word_vectorizer, mnb, 'pt')
print
most_informative_feature_for_class_svm(word_vectorizer, svcc)
```
with output:
```
pt teve -4.63472898823
pt tive -4.63472898823
pt todas -4.63472898823
pt vida -4.63472898823
pt de -4.22926388012
pt foi -4.22926388012
pt mais -4.22926388012
pt me -4.22926388012
pt as -3.94158180767
pt que -3.94158180767

no 0.0204081632653
parecer 0.0204081632653
pone 0.0204081632653
por 0.0204081632653
relación 0.0204081632653
una 0.0204081632653
visto 0.0204081632653
ya 0.0204081632653
es 0.0408163265306
lo 0.0408163265306
```


Question: How to use pickle to save sklearn model I want to dump and load my Sklearn trained model using Pickle. How to do that?

Response:
Save:
```
import pickle

with open("model.pkl", "wb") as f:
    pickle.dump(model, f)
```
Load:
```
with open("model.pkl", "rb") as f:
    model = pickle.load(f)
```
In the specific case of scikit-learn, it may be better to use joblib’s replacement of pickle (dump & load), which is more efficient on objects that carry large numpy arrays internally as is often the case for fitted scikit-learn estimators: Save:
```
import joblib

joblib.dump(model, "model.joblib")
```
Load:
```
model = joblib.load("model.joblib")
```


Question: fit-transform on training data and transform on test data [duplicate] This question already has answers here: what is the difference between 'transform' and 'fit_transform' in sklearn (9 answers) Closed 6 years ago. I am having trouble understanding how exactly transform() and fit_transform() are working together. I call fit_transform() on my training data set and transform() on my test set afterwards. However if I call fit_transform() on the test set I get bad results. Can anybody give me an explanation how and why this occurs?

Response:
Let's take an example of a transform, sklearn.preprocessing.StandardScaler. From the docs, this will: Standardize features by removing the mean and scaling to unit variance Suppose you're working with code like the following.
```
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# X is features, y is label

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=42
)
```
When you call StandardScaler.fit(X_train), what it does is calculate the mean and variance from the values in X_train. Then calling .transform() will transform all of the features by subtracting the mean and dividing by the variance. For convenience, these two function calls can be done in one step using fit_transform(). The reason you want to fit the scaler using only the training data is because you don't want to bias your model with information from the test data. If you fit() to your test data, you'd compute a new mean and variance for each feature. In theory these values may be very similar if your test and train sets have the same distribution, but in practice this is typically not the case. Instead, you want to only transform the test data by using the parameters computed on the training data.


Question: How to perform logistic lasso in python? The scikit-learn package provides the functions Lasso() and LassoCV() but no option to fit a logistic function instead of a linear one...How to perform logistic lasso in python?

Response:
The Lasso optimizes a least-square problem with a L1 penalty. By definition you can't optimize a logistic function with the Lasso. If you want to optimize a logistic function with a L1 penalty, you can use the LogisticRegression estimator with the L1 penalty:
```
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
log = LogisticRegression(penalty='l1', solver='liblinear')
log.fit(X, y)
```
Note that only the LIBLINEAR and SAGA (added in v0.19) solvers handle the L1 penalty.


Question:
Not Fitted Error when using Sklearn's graphviz when I am trying to export a random forest graph using the following command:
```
tree.export_graphviz(rnd_clf, out_file = None, feature_names = X_test[::1])
```
I receive the following error:
```
NotFittedError: This RandomForestClassifier instance is not fitted yet.
Call 'fit' with appropriate arguments before using this method.
```
What I don't understand is why it keeps telling me this, even though I have fitted the random forest classifier using:
```
rnd_clf = RandomForestClassifier(
             n_estimators=120,
             criterion='gini',
             max_features= None,
             max_depth = 14 )

rnd_clf.fit(X_train, y_train)
```
and it works perfectly fine.

Response:
(Only going by the docs; no personal experience) You are trying to plot some DecisionTree, using a function which signature reads:
```
sklearn.tree.export_graphviz(decision_tree, ...)
```
but you are passing a RandomForest, which is an ensemble of trees. That's not going to work! Going deeper, the code internally for this is here:
```
check_is_fitted(decision_tree, 'tree_')
```
So this is asking for the attribute tree_ of your DecisionTree, which exists for a DecisionTreeClassifier. This attribute does not exist for a RandomForestClassifier! Therefore the error. The only thing you can do: print every DecisionTree within your RandomForest ensemble. For this, you need to traverse random_forest.estimators_ to get the underlying decision-trees!


Question:
How to get odds-ratios and other related features with scikit-learn I'm going through this odds ratios in logistic regression tutorial, and trying to get the exactly the same results with the logistic regression module of scikit-learn. With the code below, I am able to get the coefficient and intercept but I could not find a way to find other properties of the model listed in the tutorial such as log-likelyhood, Odds Ratio, Std. Err., z, P>|z|, [95% Conf. Interval]. If someone could show me how to have them calculated with sklearn package, I would appreciate it.
```
import pandas as pd
from sklearn.linear_model import LogisticRegression

url = 'https://stats.idre.ucla.edu/wp-content/uploads/2016/02/sample.csv'
df = pd.read_csv(url, na_values=[''])
y = df.hon.values
X = df.math.values
y = y.reshape(200,1)
X = X.reshape(200,1)
clf = LogisticRegression(C=1e5)
clf.fit(X,y)
clf.coef_
clf.intercept_
```

Response:
You can get the odds ratios by taking the exponent of the coeffecients:
```
import numpy as np
X = df.female.values.reshape(200,1)
clf.fit(X,y)
np.exp(clf.coef_)

# array([[ 1.80891307]])
```
As for the other statistics, these are not easy to get from scikit-learn (where model evaluation is mostly done using cross-validation), if you need them you're better off using a different library such as statsmodels.


Question:
How to calculate a partial Area Under the Curve (AUC) In scikit learn you can compute the area under the curve for a binary classifier with
```
roc_auc_score( Y, clf.predict_proba(X)[:,1] )
```
I am only interested in the part of the curve where the false positive rate is less than 0.1. Given such a threshold false positive rate, how can I compute the AUC only for the part of the curve up the threshold? Here is an example with several ROC-curves, for illustration: The scikit learn docs show how to use roc_curve
```

```python
import numpy as np
```

```python
from sklearn import metrics
```

```python
y = np.array([1, 1, 2, 2])
```

```python
scores = np.array([0.1, 0.4, 0.35, 0.8])
```

```python
fpr, tpr, thresholds = metrics.roc_curve(y, scores, pos_label=2)
```

```python
fpr
#Output
#array([ 0. ,  0.5,  0.5,  1. ])
```

```python
tpr
#Output
#array([ 0.5,  0.5,  1. ,  1. ])
```

```python
thresholds
#Output
#array([ 0.8 ,  0.4 ,  0.35,  0.1 ]
#```
#Is there a simple way to go from this to the partial AUC? It seems the only problem is how to compute the tpr value at fpr = 0.1 as roc_curve doesn't necessarily give you that.
```

Response:
Say we start with
```
import numpy as np
from sklearn import  metrics
```
Now we set the true y and predicted scores:
```
y = np.array([0, 0, 1, 1])

scores = np.array([0.1, 0.4, 0.35, 0.8])
```
(Note that y has shifted down by 1 from your problem. This is inconsequential: the exact same results (fpr, tpr, thresholds, etc.) are obtained whether predicting 1, 2 or 0, 1, but some sklearn.metrics functions are a drag if not using 0, 1.) Let's see the AUC here:
```

```python
metrics.roc_auc_score(y, scores)
#Output
#0.75
#```
#As in your example:
#```
#fpr, tpr, thresholds = metrics.roc_curve(y, scores)
```

```python
fpr, tpr
#Output
#(array([ 0. ,  0.5,  0.5,  1. ]), array([ 0.5,  0.5,  1. ,  1. ]))
#```
#This gives the following plot:
#```
#plot([0, 0.5], [0.5, 0.5], [0.5, 0.5], [0.5, 1], [0.5, 1], [1, 1]);
#```
#By construction, the ROC for a finite-length y will be composed of rectangles: For low enough threshold, everything will be classified as negative. As the threshold increases continuously, at discrete points, some negative classifications will be changed to positive. So, for a finite y, the ROC will always be characterized by a sequence of connected horizontal and vertical lines leading from (0, 0) to (1, 1). The AUC is the sum of these rectangles. Here, as shown above, the AUC is 0.75, as the rectangles have areas 0.5 * 0.5 + 0.5 * 1 = 0.75. In some cases, people choose to calculate the AUC by linear interpolation. Say the length of y is much larger than the actual number of points calculated for the FPR and TPR. Then, in this case, a linear interpolation is an approximation of what the points in between might have been. In some cases people also follow the conjecture that, had y been large enough, the points in between would be interpolated linearly. sklearn.metrics does not use this conjecture, and to get results consistent with sklearn.metrics, it is necessary to use rectangle, not trapezoidal, summation. Let's write our own function to calculate the AUC directly from fpr and tpr:
#```
#import itertools
#import operator
```

def auc_from_fpr_tpr(fpr, tpr, trapezoid=False):
    inds = [i for (i, (s, e)) in enumerate(zip(fpr[: -1], fpr[1: ])) if s != e] + [len(fpr) - 1]
    fpr, tpr = fpr[inds], tpr[inds]
    area = 0
    ft = zip(fpr, tpr)
    for p0, p1 in zip(ft[: -1], ft[1: ]):
        area += (p1[0] - p0[0]) * ((p1[1] + p0[1]) / 2 if trapezoid else p0[1])
    return area
```
This function takes the FPR and TPR, and an optional parameter stating whether to use trapezoidal summation. Running it, we get:
```

```python
auc_from_fpr_tpr(fpr, tpr), auc_from_fpr_tpr(fpr, tpr, True)
#Output
#(0.75, 0.875)
#```
#We get the same result as sklearn.metrics for the rectangle summation, and a different, higher, result for trapezoid summation. So, now we just need to see what would happen to the FPR/TPR points if we would terminate at an FPR of 0.1. We can do this with the bisect module
#```
#import bisect
```

def get_fpr_tpr_for_thresh(fpr, tpr, thresh):
    p = bisect.bisect_left(fpr, thresh)
    fpr = fpr.copy()
    fpr[p] = thresh
    return fpr[: p + 1], tpr[: p + 1]
```
How does this work? It simply checks where would be the insertion point of thresh in fpr. Given the properties of the FPR (it must start at 0), the insertion point must be in a horizontal line. Thus all rectangles before this one should be unaffected, all rectangles after this one should be removed, and this one should be possibly shortened. Let's apply it:
```
fpr_thresh, tpr_thresh = get_fpr_tpr_for_thresh(fpr, tpr, 0.1)

```python
fpr_thresh, tpr_thresh
#Output
#(array([ 0. ,  0.1]), array([ 0.5,  0.5]))
#```
#Finally, we just need to calculate the AUC from the updated versions:
#```
```

```python
auc_from_fpr_tpr(fpr, tpr), auc_from_fpr_tpr(fpr, tpr, True)
#Output
#0.050000000000000003, 0.050000000000000003)
#```
#In this case, both the rectangle and trapezoid summations give the same results. Note that in general, they will not. For consistency with sklearn.metrics, the first one should be used.
```