Question:
How to traverse a tree from sklearn AgglomerativeClustering? I have a numpy text file array at: https://github.com/alvations/anythingyouwant/blob/master/WN_food.matrix It's a distance matrix between terms and each other, my list of terms are as such: http://pastebin.com/2xGt7Xjh I used the follow code to generate a hierarchical cluster:
```
import numpy as np
from sklearn.cluster import AgglomerativeClustering

matrix = np.loadtxt('WN_food.matrix')
n_clusters = 518
model = AgglomerativeClustering(n_clusters=n_clusters,
                                linkage="average", affinity="cosine")
model.fit(matrix)
```
To get the clusters for each term, I could have done:
```
for term, clusterid in enumerate(model.labels_):
    print term, clusterid
```
But how do I traverse the tree that the AgglomerativeClustering outputs? Is it possible to convert it into a scipy dendrogram (http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.cluster.hierarchy.dendrogram.html)? And after that how do I traverse the dendrogram?

Response:
I've answered a similar question for sklearn.cluster.ward_tree: How do you visualize a ward tree from sklearn.cluster.ward_tree? AgglomerativeClustering outputs the tree in the same way, in the children_ attribute. Here's an adaptation of the code in the ward tree question for AgglomerativeClustering. It outputs the structure of the tree in the form (node_id, left_child, right_child) for each node of the tree.
```
import numpy as np
from sklearn.cluster import AgglomerativeClustering
import itertools

X = np.concatenate([np.random.randn(3, 10), np.random.randn(2, 10) + 100])
model = AgglomerativeClustering(linkage="average", affinity="cosine")
model.fit(X)

ii = itertools.count(X.shape[0])
[{'node_id': next(ii), 'left': x[0], 'right':x[1]} for x in model.children_]
```
https://stackoverflow.com/a/26152118


Question:
Sklearn kmeans equivalent of elbow method Let's say I'm examining up to 10 clusters, with scipy I usually generate the 'elbow' plot as follows:
```
from scipy import cluster
cluster_array = [cluster.vq.kmeans(my_matrix, i) for i in range(1,10)]

pyplot.plot([var for (cent,var) in cluster_array])
pyplot.show()
```
I have since became motivated to use sklearn for clustering, however I'm not sure how to create the array needed to plot as in the scipy case. My best guess was:
```
from sklearn.cluster import KMeans

km = [KMeans(n_clusters=i) for i range(1,10)]
cluster_array = [km[i].fit(my_matrix)]
```
That unfortunately resulted in an invalid command error. What is the best way sklearn way to go about this? Thank you

Response:
you can use the inertia attribute of Kmeans class. Assuming X is your dataset:
```
from sklearn.cluster import KMeans
from matplotlib import pyplot as plt

X = # <your_data>
distorsions = []
for k in range(2, 20):
    kmeans = KMeans(n_clusters=k)
    kmeans.fit(X)
    distorsions.append(kmeans.inertia_)

fig = plt.figure(figsize=(15, 5))
plt.plot(range(2, 20), distorsions)
plt.grid(True)
plt.title('Elbow curve')
```


Question:
Sklearn transform error: Expected 2D array, got 1D array instead I use sklearn to transform data with this code.
```
sc = MinMaxScaler()

test= df['outcome']
y = sc.fit_transform(test)
```
It show error like this.
```
ValueError: Expected 2D array, got 1D array instead:
array=[ 21000. 36000.  5000. ...  7000.  12000.  11000.].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
```
How to fix it?

Response: If I remember correctly, MinMaxScalar can accept a pandas dataframe but not a series, so just do test = df[['outcome']] (dataframe with one column) instead of test = df['outcome'] (a series).


Question:
How to get the number of components needed in PCA with all extreme variance? I am trying to get the number of components needed to be used for classification. I have read a similar question Finding the dimension with highest variance using scikit-learn PCA and the scikit documents about this: http://scikit-learn.org/dev/tutorial/statistical_inference/unsupervised_learning.html#principal-component-analysis-pca However, this still did not solve my question. All of my PCA components are super big and of cause I could select all of them but if I do so PCA will be useless. I also read the PCA library in scikit learn http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html It indicates thatL: if n_components == ‘mle’, Minka’s MLE is used to guess the dimension if 0 < n_components < 1, select the number of components such that the amount of variance that needs to be explained is greater than the percentage specified by n_components However I cannot find any more information about use this techniques for analysis n_components of PCA Here is my code of PCA analysis:
```
from sklearn.decomposition import PCA
    pca = PCA()
    pca.fit(x_array_train)
    print(pca.explained_variance_)
```
result:
```
[  6.58902714e+50   6.23266555e+49   2.93568652e+49   2.25418736e+49
       1.10063872e+49   3.25107359e+40   4.72113817e+39   1.40411862e+39
       4.03270198e+38   1.60662882e+38   3.20028861e+28   2.35570241e+27
       1.54944915e+27   8.05181151e+24   1.42231553e+24   5.05155955e+23
       2.90909468e+23   2.60339206e+23   1.95672973e+23   1.22987336e+23
       9.67133111e+22   7.07208772e+22   4.49067983e+22   3.57882593e+22
       3.03546737e+22   2.38077950e+22   2.18424235e+22   1.79048845e+22
       1.50871735e+22   1.35571453e+22   1.26605081e+22   1.04851395e+22
       8.88191944e+21   6.91581346e+21   5.43786989e+21   5.05544020e+21
       4.33110823e+21   3.18309135e+21   3.06169368e+21   2.66513522e+21
       2.57173046e+21   2.36482212e+21   2.32203521e+21   2.06033130e+21
       1.89039408e+21   1.51882514e+21   1.29284842e+21   1.26103770e+21
       1.22012185e+21   1.07857244e+21   8.55143095e+20   4.82321416e+20
       2.98301261e+20   2.31336276e+20   1.31712446e+20   1.05253795e+20
       9.84992112e+19   8.27574150e+19   4.66007620e+19   4.09687463e+19
       2.89855823e+19   2.79035170e+19   1.57015298e+19   1.39218538e+19
       1.00594159e+19   7.31960049e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.29043685e+18   5.29043685e+18   5.29043685e+18
       5.29043685e+18   5.24952686e+18   2.09685699e+18   4.16588190e+17]
```
I tried PCA(n_components = 'mle') however I got these errors ..
```
Traceback (most recent call last):
  File "xx", line 166, in <module>
    pca.fit(x_array_train)
  File "xx", line 225, in fit
    self._fit(X)
  File "/Users/lib/python2.7/site-packages/sklearn/decomposition/pca.py", line 294, in _fit
    n_samples, n_features)
  File "/Users/lib/python2.7/site-packages/sklearn/decomposition/pca.py", line 98, in _infer_dimension_
    ll[rank] = _assess_dimension_(spectrum, rank, n_samples, n_features)
  File "/Users/lib/python2.7/site-packages/sklearn/decomposition/pca.py", line 83, in _assess_dimension_
    (1. / spectrum_[j] - 1. / spectrum_[i])) + log(n_samples)
ValueError: math domain error
```
Really appreciate for any helps...

Response: I'm just learning this myself, but it seems to me that the reference to using 0 < n_components < 1 suggests that you could set n_components to, say, 0.85, and the exact number of components you need to explain 85% of the variance will be used. You can verify that the right number of components is selected by also printing sum(pca.explained_variance_). You should get the smallest sum of variance percentages over 0.85 (or whatever value you chose) that's possible for your data. There are more sophisticated ways to choose a number of components, of course, but a rule of thumb of 70% - 90% is a reasonable start.


Question:
Is the xgboost documentation wrong ? (early stopping rounds and best and last iteration) here below is a question about xgboost early stopping rounds parameter and how it does, or does not, give the best iteration when it is the reason why the fit ends. In xgboost documentation, one can see in the scikit learn api section (link) that when the fit stops due to the early stopping rounds parameter: Activates early stopping. Validation error needs to decrease at least every "early_stopping_rounds" round(s) to continue training. Requires at least one item in evals. If there’s more than one, will use the last. Returns the model from the last iteration (not the best one). When reeding this, it seems that the model returned, in this case, is not the best one but the last one. To access the best one when predict, it says, it is possible to call the predict using the ntree_limit parameter with the bst.best_ntree_limit given at the end of the fit. In this sense, it should work the same way as the train of xgboost since the fit of the scikitlearn api seems to be only an embedding of the train and others. It is wiedly discussed here stack overflow discussion or here another discussion But when I tried to address this problem and check how it worked with my data, I did not find the behavior that I thought I should have. In fact the behavior I encountered was not at all the one discribed in those discussions and documentation. I call a fit this way: reg = xgb.XGBRegressor(n_jobs=6, n_estimators = 100, max_depth= 5)
```
reg.fit(
   X_train,
   y_train,
   eval_metric='rmse',
   eval_set=[(X_train, y_train), (X_valid, y_valid)],
   verbose=True,
   early_stopping_rounds = 6)
```
and here is what I get in the end:
```
[71]    validation_0-rmse:1.70071   validation_1-rmse:1.9382
[72]    validation_0-rmse:1.69806   validation_1-rmse:1.93825
[73]    validation_0-rmse:1.69732   validation_1-rmse:1.93803
Stopping. Best iteration:
[67]    validation_0-rmse:1.70768   validation_1-rmse:1.93734
```
and when I check the values of the validation I used :
```
y_pred_valid = reg.predict(X_valid)
y_pred_valid_df = pd.DataFrame(y_pred_valid)
sqrt(mse(y_valid, y_pred_valid_df[0]))
```
I get 1.9373418403889535 If the fit had return the last iteration instead of the best one it should have given an rmse around 1.93803 but it gave an rmse at 1.93734, exactly the best score. I checked again by two ways: [Edit] I've edited the code below according to @Eran Moshe answer
```
y_pred_valid = reg.predict(X_valid, ntree_limit=reg.best_ntree_limit)
y_pred_valid_df = pd.DataFrame(y_pred_valid)
sqrt(mse(y_valid, y_pred_valid_df[0]))
```
1.9373418403889535 and even if I call the fit (knowing the best iter is the 67th) with only 68 estimators so that I'm sure the last one is the best one:
```
reg = xgb.XGBRegressor(n_jobs=6, n_estimators = 68, max_depth= 5)

reg.fit(
   X_train,
   y_train,
   eval_metric='rmse',
   eval_set=[(X_train, y_train), (X_valid, y_valid)],
   verbose=True,
   early_stopping_rounds = 6)
```
the result is the same: 1.9373418403889535 So that seems to lead to the idea that, unlike the documentation, and those numerous discussions about it, tell, the fit of xgboost, when stopped by the early stopping round parameter, does give the best iter, not the last one. Am I wrong, if so, where, and how do you explain the behavior I met ? Thanks for the attention

Response: I think, it is not wrong, but inconsistent. The documentation of the predict method is correct (e.g. see here). To bee 100% sure it is better to look into the code: xgb github, so predict behaves as is stated in it's documentation, but the fit documentation is outdated. Please, post it as an issue on XGB github and either they will fix the docs or you will and will become an XGB contributer :)


Question: Best Machine Learning package for Python 3x? [closed] Closed. This question is opinion-based. It is not currently accepting answers. Want to improve this question? Because this question may lead to opinionated discussion, debate, and answers, it has been closed. You may edit the question if you feel you can improve it so that it requires answers that include facts and citations or a detailed explanation of the proposed solution. If edited, the question will be reviewed and might be reopened. Closed 5 years ago. Improve this question I was bummed out to see that scikit-learn does not support Python 3...Is there a comparable package anyone can recommend for Python 3?

Response: Try either of these, both are supported by python 3 http://mdp-toolkit.sourceforge.net/ http://mlpy.sourceforge.net/


Question:
Cannot understand with sklearn's PolynomialFeatures Need help in sklearn's Polynomial Features. It works quite well with one feature but whenever I add multiple features, it also outputs some values in the array besides the values raised to the power of the degrees. For ex: For this array,
```
X=np.array([[230.1,37.8,69.2]])
```
when I try to
```
X_poly=poly.fit_transform(X)
```
It outputs
```
[[ 1.00000000e+00 2.30100000e+02 3.78000000e+01 6.92000000e+01
5.29460100e+04 8.69778000e+03 1.59229200e+04 1.42884000e+03
2.61576000e+03 4.78864000e+03]]
```
Here, what is 8.69778000e+03,1.59229200e+04,2.61576000e+03 ?

Response:
If you have features [a, b, c] the default polynomial features(in sklearn the degree is 2) should be [1, a, b, c, a^2, b^2, c^2, ab, bc, ca]. 2.61576000e+03 is 37.8x62.2=2615,76 (2615,76 = 2.61576000 x 10^3) In a simple way with the PolynomialFeatures you can create new features. There is a good reference here. Of course there are and disadvantages("Overfitting") of using PolynomialFeatures(see here). Edit: We have to be careful when using the polynomial features. The formula for calculating the number of the polynomial features is N(n,d)=C(n+d,d) where n is the number of the features, d is the degree of the polynomial, C is binomial coefficient(combination). In our case the number is C(3+2,2)=5!/(5-2)!2!=10 but when the number of features or the degree is height the polynomial features becomes too many. For example:
```
N(100,2)=5151
N(100,5)=96560646
```
So in this case you may need to apply regularization to penalize some of the weights. It is quite possible that the algorithm will start to suffer from curse of dimensionality (here is also a very nice discussion).


Question: What is the difference between partial fit and warm start? Context: I am using Passive Aggressor from scikit library and confused whether to use warm start or partial fit. Efforts hitherto: Referred this thread discussion: https://github.com/scikit-learn/scikit-learn/issues/1585 Gone through the scikit code for _fit and _partial_fit. My observations: _fit in turn calls _partial_fit. When warm_start is set, _fit calls _partial_fit with self.coef_ When _partial_fit is called without coef_init parameter and self.coef_ is set, it continues to use self.coef_ Question: I feel both are ultimately providing the same functionalities.Then, what is the basic difference between them? In which contexts, either of them are used? Am I missing something evident? Any help is appreciated!

Response:
I don't know about the Passive Aggressor, but at least when using the SGDRegressor, partial_fit will only fit for 1 epoch, whereas fit will fit for multiple epochs (until the loss converges or max_iter is reached). Therefore, when fitting new data to your model, partial_fit will only correct the model one step towards the new data, but with fit and warm_start it will act as if you would combine your old data and your new data together and fit the model once until convergence. Example:
```
from sklearn.linear_model import SGDRegressor
import numpy as np

np.random.seed(0)
X = np.linspace(-1, 1, num=50).reshape(-1, 1)
Y = (X * 1.5 + 2).reshape(50,)

modelFit = SGDRegressor(learning_rate="adaptive", eta0=0.01, random_state=0, verbose=1,
                     shuffle=True, max_iter=2000, tol=1e-3, warm_start=True)
modelPartialFit = SGDRegressor(learning_rate="adaptive", eta0=0.01, random_state=0, verbose=1,
                     shuffle=True, max_iter=2000, tol=1e-3, warm_start=False)
# first fit some data
modelFit.fit(X, Y)
modelPartialFit.fit(X, Y)
# for both: Convergence after 50 epochs, Norm: 1.46, NNZs: 1, Bias: 2.000027, T: 2500, Avg. loss: 0.000237
print(modelFit.coef_, modelPartialFit.coef_) # for both: [1.46303288]

# now fit new data (zeros)
newX = X
newY = 0 * Y

# fits only for 1 epoch, Norm: 1.23, NNZs: 1, Bias: 1.208630, T: 50, Avg. loss: 1.595492:
modelPartialFit.partial_fit(newX, newY)

# Convergence after 49 epochs, Norm: 0.04, NNZs: 1, Bias: 0.000077, T: 2450, Avg. loss: 0.000313:
modelFit.fit(newX, newY)

print(modelFit.coef_, modelPartialFit.coef_) # [0.04245779] vs. [1.22919864]
newX = np.reshape([2], (-1, 1))
print(modelFit.predict(newX), modelPartialFit.predict(newX)) # [0.08499296] vs. [3.66702685]
```


Question:
What does KFold in python exactly do? I am looking at this tutorial: https://www.dataquest.io/mission/74/getting-started-with-kaggle I got to part 9, making predictions. In there there is some data in a dataframe called titanic, which is then divided up in folds using:
```
# Generate cross validation folds for the titanic dataset.  It return the row indices corresponding to train and test.
# We set random_state to ensure we get the same splits every time we run this.
kf = KFold(titanic.shape[0], n_folds=3, random_state=1)
```
I am not sure what is it exactly doing and what kind of object kf is. I tried reading the documentation but it did not help much. Also, there are three folds (n_folds=3), why is it later only accessing train and test (and how do I know they are called train and test) in this line?
```
for train, test in kf:
```

Response:
KFold will provide train/test indices to split data in train and test sets. It will split dataset into k consecutive folds (without shuffling by default).Each fold is then used a validation set once while the k - 1 remaining folds form the training set (source). Let's say, you have some data indices from 1 to 10. If you use n_fold=k, in first iteration you will get i'th (i<=k) fold as test indices and remaining (k-1) folds (without that i'th fold) together as train indices. An example
```
import numpy as np
from sklearn.cross_validation import KFold

x = [1,2,3,4,5,6,7,8,9,10,11,12]
kf = KFold(12, n_folds=3)

for train_index, test_index in kf:
    print (train_index, test_index)
```
Output Fold 1: [ 4 5 6 7 8 9 10 11] [0 1 2 3] Fold 2: [ 0 1 2 3 8 9 10 11] [4 5 6 7] Fold 3: [0 1 2 3 4 5 6 7] [ 8 9 10 11] Import Update for sklearn 0.20: KFold object was moved to the sklearn.model_selection module in version 0.20. To import KFold in sklearn 0.20+ use from sklearn.model_selection import KFold. KFold current documentation source


Question: Unbalanced classification using RandomForestClassifier in sklearn I have a dataset where the classes are unbalanced. The classes are either '1' or '0' where the ratio of class '1':'0' is 5:1. How do you calculate the prediction error for each class and the rebalance weights accordingly in sklearn with Random Forest, kind of like in the following link: http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#balance

Response:
You can pass sample weights argument to Random Forest fit method
```
sample_weight : array-like, shape = [n_samples] or None
```
Sample weights. If None, then samples are equally weighted. Splits that would create child nodes with net zero or negative weight are ignored while searching for a split in each node. In the case of classification, splits are also ignored if they would result in any single class carrying a negative weight in either child node. In older version there were a preprocessing.balance_weights method to generate balance weights for given samples, such that classes become uniformly distributed. It is still there, in internal but still usable preprocessing._weights module, but is deprecated and will be removed in future versions. Don't know exact reasons for this. Update Some clarification, as you seems to be confused. sample_weight usage is straightforward, once you remember that its purpose is to balance target classes in training dataset. That is, if you have X as observations and y as classes (labels), then len(X) == len(y) == len(sample_wight), and each element of sample witght 1-d array represent weight for a corresponding (observation, label) pair. For your case, if 1 class is represented 5 times as 0 class is, and you balance classes distributions, you could use simple
```
sample_weight = np.array([5 if i == 0 else 1 for i in y])
```
assigning weight of 5 to all 0 instances and weight of 1 to all 1 instances. See link above for a bit more crafty balance_weights weights evaluation function.


Question:
Get feature and class names into decision tree using export graphviz Good Afternoon, I am working on a decision tree classifier and am having trouble visualizing it. I can output the decision tree, however I cannot get my feature or class names/labels into it. My data is in a pandas dataframe format which I then move into a numpy array and pass to the classifier. I've tried a few things, but just seem to error out on the export when I try and specify class names. Any help would be appreciated. Code is below.
```
all_inputs=df.ix[:,14:].values
all_classes=df['wic'].values

(training_inputs,
 testing_inputs,
 training_classes,
 testing_classes) = train_test_split(all_inputs, all_classes,train_size=0.75, random_state=1)

decision_tree_classifier=DecisionTreeClassifier()
decision_tree_classifier.fit(training_inputs,training_classes)

export_graphviz(decision_tree_classifier, out_file="mytree.dot",
                     feature_names=??,
                     class_names=??)
```
LIke I said, it runs fine and outputs a decision tree viz if I take out the feature_names and class_names parameters. I'd like to include them in the output though if possible and have hit a wall... Any help would be greatly appreciated! Thanks, Scott

Response:
The class names are stored in decision_tree_classifier.classes_, i.e. the classes_ attribute of your DecisionTreeClassifier instance. And the feature names should be the columns of your input dataframe. For your case you will have
```
class_names = decision_tree_classifier.classes_
feature_names = df.columns[14:]
```


Question: Extracting the trees (predictor) from random forest classifier I have a specific technical question about sklearn, random forest classifier. After fitting the data with the ".fit(X,y)" method, is there a way to extract the actual trees from the estimator object, in some common format, so the ".predict(X)" method can be implemented outside python?

Response: Yes, the trees of a forest are stored in the estimators_ attribute of the forest object. You can have a look at the implementation of the export_graphviz function to learn out to write your custom exporter: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/tree/_export.py Here is the usage doc for this function: http://scikit-learn.org/stable/modules/tree.html#classification


Question:
How to calculate dice coefficient for measuring accuracy of image segmentation in python I have an image of land cover and I segmented it using K-means clustering. Now I want to calculate the accuracy of my segmentation algorithm. I read somewhere that dice co-efficient is the substantive evaluation measure. But I am not sure how to calculate it. I use Python 2.7 Are there any other effective evaluation methods? Please give a summary or a link to a source. Thank You! Edits: I used the following code for measuring the dice similarity for my original and the segmented image but it seems to take hours to calculate:
```
for i in xrange(0,7672320):
  for j in xrange(0,3):
    dice = np.sum([seg==gt])*2.0/(np.sum(seg)+np.sum(gt)) #seg is the segmented image and gt is the original image. Both are of same size
```

Response:
Please refer to Dice similarity coefficient at wiki A sample code segment here for your reference. Please note that you need to replace k with your desired cluster since you are using k-means.
```
import numpy as np

k=1

# segmentation
seg = np.zeros((100,100), dtype='int')
seg[30:70, 30:70] = k

# ground truth
gt = np.zeros((100,100), dtype='int')
gt[30:70, 40:80] = k

dice = np.sum(seg[gt==k])*2.0 / (np.sum(seg) + np.sum(gt))

print 'Dice similarity score is {}'.format(dice)
```


Question: What are different options for objective functions available in xgboost.XGBClassifier? Apart from binary:logistic (which is the default objective function), is there any other built-in objective function that can be used in xbgoost.XGBClassifier ?

Response: That's true that binary:logistic is the default objective for XGBClassifier, but I don't see any reason why you couldn't use other objectives offered by XGBoost package. For example, you can see in sklearn.py source code that multi:softprob is used explicitly in multiclass case. Moreover, if it's really necessary, you can provide a custom objective function (details here).


Question: Understanding min_df and max_df in scikit CountVectorizer I have five text files that I input to a CountVectorizer. When specifying min_df and max_df to the CountVectorizer instance what does the min/max document frequency exactly mean? Is it the frequency of a word in its particular text file or is it the frequency of the word in the entire overall corpus (five text files)? What are the differences when min_df and max_df are provided as integers or as floats? The documentation doesn't seem to provide a thorough explanation nor does it supply an example to demonstrate the use of these two parameters. Could someone provide an explanation or example demonstrating min_df and max_df?

Response: max_df is used for removing terms that appear too frequently, also known as "corpus-specific stop words". For example: max_df = 0.50 means "ignore terms that appear in more than 50% of the documents". max_df = 25 means "ignore terms that appear in more than 25 documents". The default max_df is 1.0, which means "ignore terms that appear in more than 100% of the documents". Thus, the default setting does not ignore any terms. min_df is used for removing terms that appear too infrequently. For example: min_df = 0.01 means "ignore terms that appear in less than 1% of the documents". min_df = 5 means "ignore terms that appear in less than 5 documents". The default min_df is 1, which means "ignore terms that appear in less than 1 document". Thus, the default setting does not ignore any terms.


Question:
AttributeError: 'TfidfVectorizer' object has no attribute 'get_feature_names_out' Why do I keep on getting this error? I try other codes too, but once it uses the get_feature_names_out function it will pop out this error. Below is my code:
```
from sklearn.datasets._twenty_newsgroups import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB  # fast to train and achieves a decent F-score
from sklearn import metrics
import numpy as np

def show_top10(classifier, vectorizer, categories):
    feature_names = vectorizer.get_feature_names_out()
    for i, category in enumerate(categories):
        top10 = np.argsort(classifier.coef_[i])[-10:]
        print("%s: %s" % (category, " ".join(feature_names[top10])))

newsgroups_train = fetch_20newsgroups(subset='train')
print(list(newsgroups_train.target_names))

cats = ['alt.atheism', 'sci.space', 'rec.sport.baseball', 'rec.sport.hockey']
newsgroups_train = fetch_20newsgroups(subset='train', categories=cats)
print(list(newsgroups_train.target_names))
print(newsgroups_train.filenames.shape)

vectorizer = TfidfVectorizer()
vectors = vectorizer.fit_transform(newsgroups_train.data)
print(vectors.shape)
```

Response:
When sklearn.__version__ <= 0.24.x use following method
```
get_feature_names()
```
When sklearn.__version__ >= 1.0.x use following method
```
get_feature_names_out()
```
Ref: https://github.com/scikit-learn/scikit-learn/blob/0.24.X/sklearn/feature_extraction/text.py


Question: what is the difference between transformer and estimator in sklearn? I saw both transformer and estimator were mentioned in the sklearn documentation. Is there any difference between these two words?

Response:
The basic difference is that a: Transformer transforms the input data (X) in some ways. Estimator predicts a new value (or values) (y) by using the input data (X). Both the Transformer and Estimator should have a fit() method which can be used to train them (they learn some characteristics of the data). The signature is:
```
fit(X, y)
```
fit() does not return any value, just stores the learnt data inside the object. Here X represents the samples (feature vectors) and y is the target vector (which may have single or multiple values per corresponding sample in X). Note that y can be optional in some transformers where its not needed, but its mandatory for most estimators (supervised estimators). Look at StandardScaler for example. It needs the initial data X for finding the mean and std of the data (it learns the characteristics of X, y is not needed). Each Transformer should have a transform(X, y) function which like fit() takes the input X and returns a new transformed version of X (which generally should have same number samples but may or may not have same features). On the other hand, Estimator should have a predict(X) method which should output the predicted value of y from the given X. There will be some classes in scikit-learn which implement both transform() and predict(), like KMeans, in that case carefully reading the documentation should solve your doubts.


Question:
F1-score per class for multi-class classification I'm working on a multiclass classification problem using python and scikit-learn. Currently, I'm using the classification_report function to evaluate the performance of my classifier, obtaining reports like the following:
```

```python
print(classification_report(y_true, y_pred, target_names=target_names))
#Output
#             precision    recall  f1-score   support
```

class 0       0.50      1.00      0.67         1
    class 1       0.00      0.00      0.00         1
    class 2       1.00      0.67      0.80         3

avg / total       0.70      0.60      0.61         5
```
To do further analysis, I'm interesting in obtaining the per-class f1 score of each of the classes available. Maybe something like this:
```

```python
print(calculate_f1_score(y_true, y_pred, target_class='class 0'))
#Output
#0.67
#```
#Is there something like that available on scikit-learn?
```

Response:
Taken from the f1_score docs.
```
from sklearn.metrics import f1_score
y_true = [0, 1, 2, 0, 1, 2]
y_pred = [0, 2, 1, 0, 0, 1]

f1_score(y_true, y_pred, average=None)
```
Ouputs:
```
array([ 0.8,  0. ,  0. ])
```
Which is the scores for each class.


Question:
How to standard scale a 3D matrix? I am working on a signal classification problem and would like to scale the dataset matrix first, but my data is in a 3D format (batch, length, channels). I tried to use Scikit-learn Standard Scaler:
```
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
```
But I've got this error message: Found array with dim 3. StandardScaler expected <= 2 I think one solution would be to split the matrix by each channel in multiples 2D matrices, scale them separately and then put back in 3D format, but I wonder if there is a better solution. Thank you very much.

Response:
With only 3 line of code...
```
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train.reshape(-1, X_train.shape[-1])).reshape(X_train.shape)
X_test = scaler.transform(X_test.reshape(-1, X_test.shape[-1])).reshape(X_test.shape)
```


Question: What is the negative mean absolute error in scikit-learn? I am trying to train a model using SciKit Learn's SVM module. For the scoring, I could not find the mean_absolute_error(MAE), however, negative_mean_absolute_error(NMAE) does exist. What is the difference between these 2 metrics? Lets say I get the following results for 2 models: model 1 (NMAE = -2.6), model 2(NMAE = -3.0) Which model is better? Is it model 1? Moreover, how does the negative compare to the positive? Say the following: model 1 (NMAE = -1.7), model 2(MAE = 1.4) Here, which model is better?

Response: As its name implies, negative MAE is simply the negative of the MAE, which (MAE) is by definition a positive quantity. And since MAE is an error metric, i.e. the lower the better, negative MAE is the opposite: a value of -2.6 is better than a value of -3.0. Just remove the negative signs and treat them as MAE values (which arguably also answers your second question). Keep in mind that MAE is always available in scikit-learn as a general metric (docs).


Question:
Avoid certain parameter combinations in GridSearchCV I'm using scikit-learn's GridSearchCV to iterate over a parameter space to tune a model. Specifically, I'm using it to test different hyperparameters in a neural network. The grid is as follows:
```
params = {'num_hidden_layers': [0,1,2],
          'hidden_layer_size': [64,128,256],
          'activation': ['sigmoid', 'relu', 'tanh']}
```
The problem is that I end up running redundant models when hidden num_hidden_layers is set to 0. It will run a model with 0 hidden layers and 64 units, another with 128 units, and another with 256 units. All of these models are equivalent since there is no hidden layer. This is highly inefficient and it means I need to write more code to remove redundancy in the results. Is there a way to prevent such parameter combinations, perhaps by passing a tuple of parameters?

Response:
The sklearn documentation suggests two parameter grids. So you could do something like this:
```
param_grid = [
    {'num_hidden_layers': [1,2],
      'hidden_layer_size': [64,128,256],
      'activation': ['sigmoid', 'relu', 'tanh']},
    {'num_hidden_layers': [0],
      'hidden_layer_size': [64],
      'activation': ['sigmoid', 'relu', 'tanh']}
    ]
```


Question:
Cosine similarity between each row in a Dataframe in Python I have a DataFrame containing multiple vectors each having 3 entries. Each row is a vector in my representation. I needed to calculate the cosine similarity between each of these vectors. Converting this to a matrix representation is better or is there a cleaner approach in DataFrame itself? Here is the code that I have tried.
```
import pandas as pd
from scipy import spatial
df = pd.DataFrame([X,Y,Z]).T
similarities = df.values.tolist()

for x in similarities:
    for y in similarities:
        result = 1 - spatial.distance.cosine(x, y)
```

Response:
You can directly just use sklearn.metrics.pairwise.cosine_similarity. Demo
```
import numpy as np; import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity

df = pd.DataFrame(np.random.randint(0, 2, (3, 5)))

df
##     0  1  2  3  4
##  0  1  1  1  0  0
##  1  0  0  1  1  1
##  2  0  1  0  1  0

cosine_similarity(df)
##  array([[ 1.        ,  0.33333333,  0.40824829],
##         [ 0.33333333,  1.        ,  0.40824829],
##         [ 0.40824829,  0.40824829,  1.        ]])
```


Question:
Comparison of R, statmodels, sklearn for a classification task with logistic regression I have made some experiments with logistic regression in R, python statmodels and sklearn. While the results given by R and statmodels agree, there is some discrepency with what is returned by sklearn. I would like to understand why these results are different. I understand that it is probably not the same optimization algorithms that are used under the wood. Specifically, I use the standard Default dataset (used in the ISL book). The following Python code reads the data into a dataframe Default.
```
import pandas as pd
 # data is available here
Default = pd.read_csv('https://d1pqsl2386xqi9.cloudfront.net/notebooks/Default.csv', index_col=0)
 #
Default['default']=Default['default'].map({'No':0, 'Yes':1})
Default['student']=Default['student'].map({'No':0, 'Yes':1})
 #
I=Default['default']==0
print("Number of 'default' values :", Default[~I]['balance'].count())
```
Number of 'default' values : 333. There is a total of 10000 examples, with only 333 positives Logistic regression in R I use the following
```
library("ISLR")
data(Default,package='ISLR')
 #write.csv(Default,"default.csv")
glm.out=glm('default~balance+income+student', family=binomial, data=Default)
s=summary(glm.out)
print(s)
#
glm.probs=predict(glm.out,type="response")
glm.probs[1:5]
glm.pred=ifelse(glm.probs>0.5,"Yes","No")
 #attach(Default)
t=table(glm.pred,Default$default)
print(t)
score=mean(glm.pred==Default$default)
print(paste("score",score))
```
The result is as follows Call: glm(formula = "default~balance+income+student", family = binomial, data = Default) Deviance Residuals: Min 1Q Median 3Q Max -2.4691 -0.1418 -0.0557 -0.0203 3.7383 Coefficients:
```
Estimate Std. Error z value Pr(>|z|)
(Intercept) -1.087e+01  4.923e-01 -22.080  < 2e-16
balance      5.737e-03    2.319e-04  24.738  < 2e-16
income       3.033e-06  8.203e-06   0.370   0.71152
studentYes  -6.468e-01  2.363e-01  -2.738  0.00619
```
(Dispersion parameter for binomial family taken to be 1)
```
Null deviance: 2920.6  on 9999  degrees of freedom Residual
```
deviance: 1571.5 on 9996 degrees of freedom AIC: 1579.5 Number of Fisher Scoring iterations: 8
```
glm.pred   No  Yes
 No  9627  228
 Yes   40  105
```
1 "score 0.9732" I am too lazy to cut and paste the results obtained with statmodels. It suffice to say that they are extremely similar to those given by R. sklearn For sklearn, I ran the following code. There is a parameter class_weight for taking into account unbalanced classes. I tested class_weight=None (no weightening -- I think that is the default in R), and class_weight='auto' (weightening with the inverse frequencies found inthe data) I also put C=10000,the inverse of the regularization parameter, so as to minimize the effect of regularization. ~~
```
import sklearn
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix

features = Default[[ 'balance', 'income' ]]
target = Default['default']
#
for weight in (None,  'auto'):
    print("*"*40+"\nweight:",weight)

    classifier = LogisticRegression(C=10000, class_weight=weight, random_state=42)
                #C=10000 ~ no regularization

    classifier.fit(features, target,)  #fit classifier on whole base
    print("Intercept", classifier.intercept_)
    print("Coefficients", classifier.coef_)

    y_true=target
    y_pred_cls=classifier.predict_proba(features)[:,1]>0.5
    C=confusion_matrix(y_true,y_pred_cls)

    score=(C[0,0]+C[1,1])/(C[0,0]+C[1,1]+C[0,1]+C[1,0])
    precision=(C[1,1])/(C[1,1]+C[0 ,1])
    recall=(C[1,1])/(C[1,1]+C[1,0])
    print("\n Confusion matrix")
    print(C)
    print()
    print('{s:{c}<{n}}{num:2.4}'.format(s='Score',n=15,c='', num=score))
    print('{s:{c}<{n}}{num:2.4}'.format(s='Precision',n=15,c='', num=precision))
    print('{s:{c}<{n}}{num:2.4}'.format(s='Recall',n=15,c='', num=recall))
```
The results are given below.
```
> ****************************************
>weight: None
>
>Intercept [ -1.94164126e-06]
>
>Coefficients [[ 0.00040756 -0.00012588]]
>
>  Confusion matrix
>
>     [[9664    3]
>     [ 333    0]]
>
>     Score          0.9664
>     Precision      0.0
>     Recall         0.0
>
> ****************************************
>weight: auto
>
>Intercept [-8.15376429]
>
>Coefficients
>[[  5.67564834e-03   1.95253338e-05]]
>
>  Confusion matrix
>
>     [[8356 1311]
>     [  34  299]]
>
>     Score          0.8655
>     Precision      0.1857
>     Recall         0.8979
```
What I observe is that for class_weight=None, the Score is excellent but no positive example is recognized. Precision and recall are at zero. The coefficients found are very small, particularly the intercept. Modifying C does not change things. For class_weight='auto' things seems better but I still have a precision which is very low (too much positive classified). Again, changing C does not help. If I modify the intercept by hand, I can recover the results given by R. So I suspect that here is a discrepency between the estimation of the intecepts in the two cases. As this has a consequence in the specification of the threeshold (analog to a resampling of pulations), this can explain the differences in performances. However, I would welcome any advice for the choice between the two solutions and help to understand the origin of these differences. Thanks.

Response:
Although this post is old, I wanted to give you a solution. In your post you are comparing apples with oranges. In your R code, you are estimating "balance, income, and student" on "default". In your Python code, you are only estimating "balance and income" on "default". Of course, you cannot get the same estimates. Also the differences cannot be attributed to feature scaling, as logistic regression usually does not need it in comparison to kmeans. You are right to set a high C, so that there is no regularization. If you want to have the same output as in R, you have to change the solver to "newton-cg". Different solvers can give different results but they still yield the same objective value. As long as your solver converge everything will be okay. Here's the code that give you the same estimates like in R and Statsmodels:
```
import pandas as pd
from sklearn.linear_model import LogisticRegression
from patsy import dmatrices #
import numpy as np

 # data is available here
Default = pd.read_csv('https://d1pqsl2386xqi9.cloudfront.net/notebooks/Default.csv', index_col=0)
 #
Default['default']=Default['default'].map({'No':0, 'Yes':1})
Default['student']=Default['student'].map({'No':0, 'Yes':1})

# use dmatrices to get data frame for logistic regression
y, X = dmatrices('default ~ balance+income+C(student)',
                  Default,return_type="dataframe")

y = np.ravel(y)

# fit logistic regression
model = LogisticRegression(C = 1e6, fit_intercept=False, solver = "newton-cg", max_iter=10000000)
model = model.fit(X, y)

# examine the coefficients
pd.DataFrame(zip(X.columns, np.transpose(model.coef_)))
```


Question:
sklearn Hierarchical Agglomerative Clustering using similarity matrix Given a distance matrix, with similarity between various professors :
```
prof1     prof2     prof3
       prof1     0        0.8     0.9
       prof2     0.8      0       0.2
       prof3     0.9      0.2     0
```
I need to perform hierarchical clustering on this data, where the above data is in the form of 2-d matrix
```
data_matrix=[[0,0.8,0.9],[0.8,0,0.2],[0.9,0.2,0]]
```
I tried checking if I can implement it using sklearn.cluster AgglomerativeClustering but it is considering all the 3 rows as 3 separate vectors and not as a distance matrix. Can it be done using this or scipy.cluster.hierarchy?

Response:
Yes, you can do it with sklearn. You need to set: affinity='precomputed', to use a matrix of distances linkage='complete' or 'average', because default linkage(Ward) works only on coordinate input. With precomputed affinity, input matrix is interpreted as a matrix of distances between observations. The following code
```
from sklearn.cluster import AgglomerativeClustering
data_matrix = [[0,0.8,0.9],[0.8,0,0.2],[0.9,0.2,0]]
model = AgglomerativeClustering(affinity='precomputed', n_clusters=2, linkage='complete').fit(data_matrix)
print(model.labels_)
```
will return labels [1 0 0]: the 1st professor goes to one cluster, and the 2nd and 3rd - to another.


Question: What's the difference between predict_proba and decision_function in scikit-learn? I'm studying a scikit-learn example (Classifier comparison) and got confused with predict_proba and decision_function. They plot the classification results by drawing the contours using either Z = clf.decision_function(), or Z = clf.predict_proba(). What's the differences between these two? Is it so that each classification method has either of the two as score? Which one is more proper to interpret the classification result and how should I choose from the two?

Response: The latter, predict_proba is a method of a (soft) classifier outputting the probability of the instance being in each of the classes. The former, decision_function, finds the distance to the separating hyperplane. For example, a(n) SVM classifier finds hyperplanes separating the space into areas associated with classification outcomes. This function, given a point, finds the distance to the separators. I'd guess that predict_prob is more useful in your case, in general - the other method is more specific to the algorithm.


Question:
is there away to output selected columns names from SelectFromModel method? i performed feature selection using ExtraTreesClassifier and SelectFromModel in data set that loaded as DataFrame, however i want to save these selected feature as DataFrame to csv file while maintaining columns name as well. note that output is numpy array return important features whole columns not columns header
```
import pandas as pd
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.feature_selection import SelectFromModel
import numpy as np

df = pd.read_csv('los_10_one_encoder.csv')
y = df['LOS'] # target
X= df.drop('LOS',axis=1) # drop LOS column
clf = ExtraTreesClassifier()
clf = clf.fit(X, y)
print clf.feature_importances_

model = SelectFromModel(clf, prefit=True)
X_new = model.transform(X)
```

Response:
```
model = SelectFromModel(clf, prefit=True)
feature_idx = model.get_support()
feature_name = df.columns[feature_idx]
```


Question:
Classification metrics can't handle a mix of binary and continuous targets [duplicate] This question already has answers here: Accuracy Score ValueError: Can't Handle mix of binary and continuous target (9 answers) Closed 4 years ago. I try to train and test several scikit-learn models and attempt to print off the accuracy. Only some of these models work, others fail with the
```
ValueError: Classification metrics can't handle a mix of binary and continuous targets.
```
What does this error mean? How can I modify my code below to successfully evaluate the models that are failing?
```
import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
from sklearn import model_selection
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score

from sklearn import linear_model
from sklearn import svm

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC

from sklearn import preprocessing
from sklearn import utils

# Shuffle pandas rows randomly
from sklearn.utils import shuffle

# Disable annoying warnings
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.simplefilter(action='ignore', category=UserWarning)

# Performance
import time

# Import the data and assign the column names
colNames = []
for colName in range(0,3780):
    colNames.append("%s"%(colName))
colNames.append('class')
df = pd.read_csv("HoGTestData.csv", names=colNames)

# Randomly shuffle rows
df = shuffle(df)
df = df.head(20)

# Print some info on the dataset
print("Head of Data:")
print(df.head())
print("Shape of Data:")
print(df.shape)

# descriptions
print("Describe Data:")
#print(df.describe())

# class distribution
print(df.groupby('class').size())

# Split-out validation dataset
datasetData = df.values

# Determine shape and portion of data that is real data as opposed to labels
shape = datasetData.shape
thresh = int(shape[1]) - 1

# Extract labels and feature vectors
featureVectors = datasetData[:,0:thresh]
labels = datasetData[:,thresh:]

# Perform a standard scaler on the data
scaler = preprocessing.StandardScaler()
featureVectors = scaler.fit_transform(featureVectors)

# Encode labels to be acceptable
labelEncoder = preprocessing.LabelEncoder()
labels = labelEncoder.fit_transform(labels)

# Split data into training and testing data
test_size = 0.20
seed = 7
featureVectorTrain, featureVectorTest, labelsTrain, labelsTest = model_selection.train_test_split(featureVectors, labels, test_size=test_size, random_state=seed)

# Spot Check Algorithms
models = []
models.append(('SVM', svm.SVC()))
models.append(('LR', LogisticRegression()))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
#models.append(('SGDRegressor', linear_model.SGDRegressor())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('BayesianRidge', linear_model.BayesianRidge())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('LassoLars', linear_model.LassoLars())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('ARDRegression', linear_model.ARDRegression())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('PassiveAggressiveRegressor', linear_model.PassiveAggressiveRegressor())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('TheilSenRegressor', linear_model.TheilSenRegressor())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets
#models.append(('LinearRegression', linear_model.LinearRegression())) #ValueError: Classification metrics can't handle a mix of binary and continuous targets

# Test options and evaluation metric
seed = 42
scoring = 'accuracy'

# evaluate each model in turn
results = []
names = []
print("---------------------------------------")
for name, model in models:
    start_time = time.time()
    kfold = model_selection.KFold(n_splits=10, random_state=seed)
    cv_results = model_selection.cross_val_score(model, featureVectorTrain, labelsTrain, cv=kfold, scoring=scoring)
    elapsed_time = time.time() - start_time
    results.append(cv_results)
    names.append(name)
    msg = "{:3.2f} ({:3.2f})  Time elapsed: {:6.2f}".format(cv_results.mean(), cv_results.std(), elapsed_time)
    msg = "%s "%(name) + msg
    print(msg)
print("---------------------------------------")

print("Done")
```
Here is the script output:
```
Head of Data:
              0         1         2  ...        3778      3779  class
20573  0.124282  0.090376  0.088723  ...    0.148411  0.120542     -1
20461  0.154031  0.110177  0.087799  ...    0.100416  0.119484     -1
10416  0.340767  0.150863  0.025489  ...    0.047592  0.036171      1
52404  0.000000  0.000000  0.000000  ...    0.000000  0.000000     -1
42785  0.159105  0.118963  0.090405  ...    0.009996  0.027460     -1

[5 rows x 3781 columns]
Shape of Data:
(1024, 3781)
Describe Data:
class
-1    794
 1    230
dtype: int64
---------------------------------------
SVM 0.9878 (0.0123)  Time elapsed:  10.20
LR 0.9414 (0.0187)  Time elapsed:   7.09
LDA 0.9768 (0.0128)  Time elapsed:   6.60
KNN 0.8511 (0.0384)  Time elapsed:   3.06
CART 0.9047 (0.0358)  Time elapsed:   8.84
NB 0.9292 (0.0209)  Time elapsed:   0.36
---------------------------------------
Done
```
Here is the labelsTrain variable:
```
print(labelsTrain)
[0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 1 1 1 0 1 1 0 0
 1 1 0 1 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0
 0 0 1 1 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 1 0 0 1 0 1 0 0 1 0 0 0 1 1 0 0 0
 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 1 1 1 0 1 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1
 1 0 1 1 0 0 0 1 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 1 1 0 0 0 0 1 0 0 1 0 0 0 1
 0 1 0 1 0 0 0 0 1 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 1 0 0 0 0 0 0 1 0 0 1 0 0
 0 0 0 0 1 1 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0
 1 0 0 0 0 0 0 0 1 0 0 0 1 0 0 1 0 0 0 1 1 0 1 0 1 1 0 1 0 0 0 0 0 0 1 0 1
 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0
 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0
 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0
 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0
 0 0 0 1 1 1 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 0 0
 1 0 0 0 0 0 1 1 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0
 1 0 0 0 0 0 0 1 1 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 1 0 1
 0 0 0 0 1 0 1 0 0 1 1 1 1 0 1 0 0 0 0 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0
 0 0 1 1 0 0 1 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0
 0 0 1 0 0]
```
The errors happen during the cross_val_score function:
```
# evaluate each model in turn
results = []
names = []
print("---------------------------------------")
for name, model in models:
    start_time = time.time()
    kfold = model_selection.KFold(n_splits=10, random_state=seed)
    print("start cross_val_score")
    cv_results = model_selection.cross_val_score(model, featureVectorTrain, labelsTrain, cv=kfold, scoring=scoring)
    print("done cross_val_score")
    elapsed_time = time.time() - start_time
    results.append(cv_results)
    #print(results)
    names.append(name)
    msg = "{:3.4f} ({:3.4f})  Time elapsed: {:6.2f}".format(cv_results.mean(), cv_results.std(), elapsed_time)
    msg = "%s "%(name) + msg
    print(msg)
print("---------------------------------------")

...

---------------------------------------
start cross_val_score
done cross_val_score
SVM 0.9744 (0.0127)  Time elapsed:  10.46
start cross_val_score
done cross_val_score
LR 0.9194 (0.0390)  Time elapsed:   9.56
start cross_val_score
done cross_val_score
LDA 0.9780 (0.0106)  Time elapsed:   8.04
start cross_val_score
done cross_val_score
KNN 0.8657 (0.0319)  Time elapsed:   3.20
start cross_val_score
done cross_val_score
CART 0.9072 (0.0326)  Time elapsed:  10.20
start cross_val_score
done cross_val_score
NB 0.9182 (0.0327)  Time elapsed:   0.38
start cross_val_score
Traceback (most recent call last):
  File "/Users/me/Desktop/MachineLearning/Initial.py", line 112, in <module>
    cv_results = model_selection.cross_val_score(model, featureVectorTrain, labelsTrain, cv=kfold, scoring=scoring)
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 402, in cross_val_score
    error_score=error_score)
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 240, in cross_validate
    for train, test in cv.split(X, y, groups))
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/parallel.py", line 917, in __call__
    if self.dispatch_one_batch(iterator):
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/parallel.py", line 759, in dispatch_one_batch
    self._dispatch(tasks)
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/parallel.py", line 716, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/_parallel_backends.py", line 182, in apply_async
    result = ImmediateResult(func)
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/_parallel_backends.py", line 549, in __init__
    self.results = batch()
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/parallel.py", line 225, in __call__
    for func, args, kwargs in self.items]
  File "/usr/local/lib/python3.7/site-packages/sklearn/externals/joblib/parallel.py", line 225, in <listcomp>
    for func, args, kwargs in self.items]
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 568, in _fit_and_score
    test_scores = _score(estimator, X_test, y_test, scorer, is_multimetric)
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 605, in _score
    return _multimetric_score(estimator, X_test, y_test, scorer)
  File "/usr/local/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 635, in _multimetric_score
    score = scorer(estimator, X_test, y_test)
  File "/usr/local/lib/python3.7/site-packages/sklearn/metrics/scorer.py", line 98, in __call__
    **self._kwargs)
  File "/usr/local/lib/python3.7/site-packages/sklearn/metrics/classification.py", line 176, in accuracy_score
    y_type, y_true, y_pred = _check_targets(y_true, y_pred)
  File "/usr/local/lib/python3.7/site-packages/sklearn/metrics/classification.py", line 81, in _check_targets
    "and {1} targets".format(type_true, type_pred))
ValueError: Classification metrics can't handle a mix of binary and continuous targets
```

Response:
All your commented-out models are not classifiers but regression models, for which accuracy is meaningless. You get the error because these regression models do not produce binary outcomes, but continuous (float) numbers (as all regression models do); so, when scikit-learn attempts to calculate the accuracy by comparing a binary number (true label) with a float (predicted value), it not unexpectedly gives an error. And this cause is clearly hinted at the error message itself:
```
Classification metrics can't handle a mix of binary and continuous target
```


Question:
What is the difference between cross_val_score with scoring='roc_auc' and roc_auc_score? I am confused about the difference between the cross_val_score scoring metric 'roc_auc' and the roc_auc_score that I can just import and call directly. The documentation (http://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter) indicates that specifying scoring='roc_auc' will use the sklearn.metrics.roc_auc_score. However, when I implement GridSearchCV or cross_val_score with scoring='roc_auc' I receive very different numbers that when I call roc_auc_score directly. Here is my code to help demonstrate what I see:
```
# score the model using cross_val_score

rf = RandomForestClassifier(n_estimators=150,
                            min_samples_leaf=4,
                            min_samples_split=3,
                            n_jobs=-1)

scores = cross_val_score(rf, X, y, cv=3, scoring='roc_auc')

print scores
array([ 0.9649023 ,  0.96242235,  0.9503313 ])

# do a train_test_split, fit the model, and score with roc_auc_score

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)
rf.fit(X_train, y_train)

print roc_auc_score(y_test, rf.predict(X_test))
0.84634039111363313 # quite a bit different than the scores above!
```
I feel like I am missing something very simple here -- most likely a mistake in how I am implementing/interpreting one of the scoring metrics. Can anyone shed any light on the reason for the discrepancy between the two scoring metrics?

Response:
This is because you supplied predicted y's instead of the probability in roc_auc_score. This function takes a score, not the classified label. Try instead to do this:
```
print roc_auc_score(y_test, rf.predict_proba(X_test)[:,1])
```
It should give a similar result to previous result from cross_val_score. Refer to this post for more info.


Question: Plotting 3D Decision Boundary From Linear SVM I've fit a 3 feature data set using sklearn.svm.svc(). I can plot the point for each observation using matplotlib and Axes3D. I want to plot the decision boundary to see the fit. I've tried adapting the 2D examples for plotting the decision boundary to no avail. I understand that clf.coef_ is a vector normal to the decision boundary. How can I plot this to see where it divides the points?

Response:
Here is an example on a toy dataset. Note that plotting in 3D is funky with matplotlib. Sometimes points that are behind the plane might appear as though they are in front of it, so you may have to fiddle with rotating the plot to ascertain what's going on.
```py
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from sklearn.svm import SVC

rs = np.random.RandomState(1234)

# Generate some fake data.
n_samples = 200
# X is the input features by row.
X = np.zeros((200,3))
X[:n_samples/2] = rs.multivariate_normal( np.ones(3), np.eye(3), size=n_samples/2)
X[n_samples/2:] = rs.multivariate_normal(-np.ones(3), np.eye(3), size=n_samples/2)
# Y is the class labels for each row of X.
Y = np.zeros(n_samples); Y[n_samples/2:] = 1

# Fit the data with an svm
svc = SVC(kernel='linear')
svc.fit(X,Y)

# The equation of the separating plane is given by all x in R^3 such that:
# np.dot(svc.coef_[0], x) + b = 0. We should solve for the last coordinate
# to plot the plane in terms of x and y.

z = lambda x,y: (-svc.intercept_[0]-svc.coef_[0][0]*x-svc.coef_[0][1]*y) / svc.coef_[0][2]

tmp = np.linspace(-2,2,51)
x,y = np.meshgrid(tmp,tmp)

# Plot stuff.
fig = plt.figure()
ax  = fig.add_subplot(111, projection='3d')
ax.plot_surface(x, y, z(x,y))
ax.plot3D(X[Y==0,0], X[Y==0,1], X[Y==0,2],'ob')
ax.plot3D(X[Y==1,0], X[Y==1,1], X[Y==1,2],'sr')
plt.show()
```
Output: EDIT (Key Mathematical Linear Algebra Statement In Comment Above):
```py
# The equation of the separating plane is given by all x in R^3 such that:
# np.dot(coefficients, x_vector) + intercept_value = 0.
# We should solve for the last coordinate: x_vector[2] == z
# to plot the plane in terms of x and y.
```


Question:
sklearn plot confusion matrix with labels I want to plot a confusion matrix to visualize the classifer's performance, but it shows only the numbers of the labels, not the labels themselves:
```
from sklearn.metrics import confusion_matrix
import pylab as pl
y_test=['business', 'business', 'business', 'business', 'business', 'business', 'business', 'business', 'business', 'business', 'business', 'business', 'business', 'business', 'business', 'business', 'business', 'business', 'business', 'business']

pred=array(['health', 'business', 'business', 'business', 'business',
       'business', 'health', 'health', 'business', 'business', 'business',
       'business', 'business', 'business', 'business', 'business',
       'health', 'health', 'business', 'health'],
      dtype='|S8')

cm = confusion_matrix(y_test, pred)
pl.matshow(cm)
pl.title('Confusion matrix of the classifier')
pl.colorbar()
pl.show()
```
How can I add the labels (health, business..etc) to the confusion matrix?

Response:
UPDATE: Check the ConfusionMatrixDisplay OLD ANSWER: I think it's worth mentioning the use of seaborn.heatmap here.
```
import seaborn as sns
import matplotlib.pyplot as plt

ax= plt.subplot()
sns.heatmap(cm, annot=True, fmt='g', ax=ax);  #annot=True to annotate cells, ftm='g' to disable scientific notation

# labels, title and ticks
ax.set_xlabel('Predicted labels');ax.set_ylabel('True labels');
ax.set_title('Confusion Matrix');
ax.xaxis.set_ticklabels(['business', 'health']); ax.yaxis.set_ticklabels(['health', 'business']);
```


Question: How is the R2 value in Scikit learn calculated? The R^2 value returned by scikit learn (metrics.r2_score()) can be negative. The docs say: "Unlike most other scores, R² score may be negative (it need not actually be the square of a quantity R)." However the wikipedia article on R^2 mentions no R (not squared) quantity. Perhaps it uses absolute differences instead of square differences. I really have no idea

Response: The R^2 in scikit learn is essentially the same as what is described in the wikipedia article on the coefficient of determination (grep for "the most general definition"). It is 1 - residual sum of square / total sum of squares. The big difference between a classical stats setting and what you usually try to do with machine learning, is that in machine learning you evaluate your score on unseen data, which can lead to results outside [0,1]. If you apply R^2 to the same data you used to fit your model, it will lie within [0, 1] See also this very similar question


Question:
List the words in a vocabulary according to occurrence in a text corpus, with Scikit-Learn CountVectorizer I have fitted a CountVectorizer to some documents in scikit-learn. I would like to see all the terms and their corresponding frequency in the text corpus, in order to select stop-words. For example
```none
'and' 123 times, 'to' 100 times, 'for' 90 times, ... and so on
```
Is there any built-in function for this?

Response:
If cv is your CountVectorizer and X is the vectorized corpus, then
```
zip(cv.get_feature_names(),
    np.asarray(X.sum(axis=0)).ravel())
```
returns a list of (term, frequency) pairs for each distinct term in the corpus that the CountVectorizer extracted. (The little asarray + ravel dance is needed to work around some quirks in scipy.sparse.)


Question:
Adding New Text to Sklearn TFIDIF Vectorizer (Python) Is there a function to add to the existing corpus? I've already generated my matrix, I'm looking to periodically add to the table without re-crunching the whole sha-bang e.g;
```
articleList = ['here is some text blah blah','another text object', 'more foo for your bar right now']
tfidf_vectorizer = TfidfVectorizer(
                        max_df=.8,
                        max_features=2000,
                        min_df=.05,
                        preprocessor=prep_text,
                        use_idf=True,
                        tokenizer=tokenize_text
                    )
tfidf_matrix = tfidf_vectorizer.fit_transform(articleList)

#### ADDING A NEW ARTICLE TO EXISTING SET?
bigger_tfidf_matrix = tfidf_vectorizer.fit_transform(['the last article I wanted to add'])
```

Response:
You can access the vocabulary_ attribute of your vectoriser directly, and you can access the idf_ vector via _tfidf._idf_diag, so it would be possible to monkey-patch something like this:
```
import re
import numpy as np
from scipy.sparse.dia import dia_matrix
from sklearn.feature_extraction.text import TfidfVectorizer

def partial_fit(self, X):
    max_idx = max(self.vocabulary_.values())
    for a in X:
        #update vocabulary_
        if self.lowercase: a = a.lower()
        tokens = re.findall(self.token_pattern, a)
        for w in tokens:
            if w not in self.vocabulary_:
                max_idx += 1
                self.vocabulary_[w] = max_idx

        #update idf_
        df = (self.n_docs + self.smooth_idf)/np.exp(self.idf_ - 1) - self.smooth_idf
        self.n_docs += 1
        df.resize(len(self.vocabulary_))
        for w in tokens:
            df[self.vocabulary_[w]] += 1
        idf = np.log((self.n_docs + self.smooth_idf)/(df + self.smooth_idf)) + 1
        self._tfidf._idf_diag = dia_matrix((idf, 0), shape=(len(idf), len(idf)))

TfidfVectorizer.partial_fit = partial_fit
articleList = ['here is some text blah blah','another text object', 'more foo for your bar right now']
vec = TfidfVectorizer()
vec.fit(articleList)
vec.n_docs = len(articleList)
vec.partial_fit(['the last text I wanted to add'])
vec.transform(['the last text I wanted to add']).toarray()

# array([[ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
#          0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
#          0.        ,  0.        ,  0.27448674,  0.        ,  0.43003652,
#          0.43003652,  0.43003652,  0.43003652,  0.43003652]])
```


Question:
sklearn: calculating accuracy score of k-means on the test data set I am doing k-means clustering on the set of 30 samples with 2 clusters (I already know there are two classes). I divide my data into training and test set and try to calculate the accuracy score on my test set. But there are two problems: first I don't know if I can actually do this (accuracy score on test set) for k-means clustering. Second: if I am allowed to do this, whether my implementation is right or wrong. Here is what I've tried:
```
df_hist = pd.read_csv('video_data.csv')

y = df_hist['label'].values
del df_hist['label']
df_hist.to_csv('video_data1.csv')
X = df_hist.values.astype(np.float)

X_train, X_test,y_train,y_test =  cross_validation.train_test_split(X,y,test_size=0.20,random_state=70)
k_means = cluster.KMeans(n_clusters=2)
k_means.fit(X_train)
print(k_means.labels_[:])
print(y_train[:])

score = metrics.accuracy_score(y_test,k_means.predict(X_test))
print('Accuracy:{0:f}'.format(score))

k_means.predict(X_test)
print(k_means.labels_[:])
print(y_test[:])
```
But, when I print k-means labels for the test set (k_means.predict(X_test) print(k_means.labels_[:])) and y_test labels (print(k_means.labels_[:])) in the last three lines, I get the same label as the ones when I was fitting the the X-train, rather than the labels that were produced for the X-test. Any idea what I might be doing wrong here? Is it right at all what I'm doing to evaluate the performance of k-means? Thank you!

Response:
In terms of evaluating accuracy. You should remember that k-means is not a classification tool, thus analyzing accuracy is not a very good idea. You can do this, but this is not what k-means is for. It is supposed to find a grouping of data which maximizes between-clusters distances, it does not use your labeling to train. Consequently, things like k-means are usually tested with things like RandIndex and other clustering metrics. For maximization of accuracy you should fit actual classifier, like kNN, logistic regression, SVM, etc. In terms of the code itself, k_means.predict(X_test) returns labeling, it does not update the internal labels_ field, you should do
```
print(k_means.predict(X_test))
```
Furthermore in python you do not have to (and should not) use [:] to print an array, just do
```
print(k_means.labels_)
print(y_test)
```


Question:
RandomForestRegressor and feature_importances_ error I am struggling to pull out the feature importances from my RandomForestRegressor, I get an: AttributeError: 'GridSearchCV' object has no attribute 'feature_importances_'. Anyone know why there is no attribute? According to documentation there should exist this attribute? The full code:
```
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import GridSearchCV

#Running a RandomForestRegressor GridSearchCV to tune the model.
parameter_candidates = {
    'n_estimators' : [650, 700, 750, 800],
    'min_samples_leaf' : [1, 2, 3],
    'max_depth' : [10, 11, 12],
    'min_samples_split' : [2, 3, 4, 5, 6]
}

RFR_regr = RandomForestRegressor()
CV_RFR_regr = GridSearchCV(estimator=RFR_regr, param_grid=parameter_candidates, n_jobs=5, verbose=2)
CV_RFR_regr.fit(X_train, y_train)

#Predict with testing set
y_pred = CV_RFR_regr.predict(X_test)

#Extract feature importances
importances = CV_RFR_regr.feature_importances_
```

Response:
You are trying to use the attribute on the GridSearchCV object. Its not present there. What you actually need to do is to access the estimator on which the grid search is done. Access the attribute by :
```
importances = CV_RFR_regr.best_estimator_.feature_importances_
```


Question:
Adjust size of ConfusionMatrixDisplay (ScikitLearn) How to set the size of the figure ploted by ScikitLearn's Confusion Matrix?
```py
import numpy as np
from sklearn.metrics import ConfusionMatrixDisplay, confusion_matrix
cm = confusion_matrix(np.arange(25), np.arange(25))
cmp = ConfusionMatrixDisplay(cm, display_labels=np.arange(25))
cmp.plot()
```
The code above shows this figure, which is too tight:

Response:
You can send a matplotlib.axes object to the .plot method of sklearn.metrics.ConfusionMatrixDisplay. Set the size of the figure in matplotlib.pyplot.subplots first.
```py
import numpy as np
from sklearn.metrics import ConfusionMatrixDisplay, confusion_matrix
import matplotlib.pyplot as plt

cm = confusion_matrix(np.arange(25), np.arange(25))
cmp = ConfusionMatrixDisplay(cm, display_labels=np.arange(25))
fig, ax = plt.subplots(figsize=(10,10))
cmp.plot(ax=ax)
```


Question: How to set k-Means clustering labels from highest to lowest with Python? I have a dataset of 38 apartments and their electricity consumption in the morning, afternoon and evening. I am trying to clusterize this dataset using the k-Means implementation from scikit-learn, and am getting some interesting results. First clustering results: This is all very well, and with 4 clusters I obviously get 4 labels associated to each apartment - 0, 1, 2 and 3. Using the random_state parameter of KMeans method, I can fix the seed in which the centroids are randomly initialized, so consistently I get the same labels attributed to the same apartments. However, as this specific case is in regards of energy consumption, a measurable classification between the highest and the lowest consumers can be performed. I would like, thus, to assign the label 0 to the apartments with lowest consumption level, label 1 to apartments that consume a bit more and so on. As of now, my labels are [2 1 3 0], or ["black", "green", "blue", "red"]; I would like them to be [0 1 2 3] or ["red", "green", "black", "blue"]. How should I proceed to do so, while still keeping the centroid initialization random (with fixed seed)? Thank you very much for the help!

Response:
Transforming the labels through a lookup table is a straightforward way to achieve what you want. To begin with I generate some mock data:
```
import numpy as np

np.random.seed(1000)

n = 38
X_morning = np.random.uniform(low=.02, high=.18, size=38)
X_afternoon = np.random.uniform(low=.05, high=.20, size=38)
X_night = np.random.uniform(low=.025, high=.175, size=38)
X = np.vstack([X_morning, X_afternoon, X_night]).T
```
Then I perform clustering on data:
```
from sklearn.cluster import KMeans
k = 4
kmeans = KMeans(n_clusters=k, random_state=0).fit(X)
```
And finally I use NumPy's argsort to create a lookup table like this:
```
idx = np.argsort(kmeans.cluster_centers_.sum(axis=1))
lut = np.zeros_like(idx)
lut[idx] = np.arange(k)
```
Sample run:
```

```python
kmeans.cluster_centers_.sum(axis=1)
#Output
#array([ 0.3214523 ,  0.40877735,  0.26911353,  0.25234873])
```

```python
idx
#Output
#array([3, 2, 0, 1], dtype=int64)
```

```python
lut
#Output
#array([2, 3, 1, 0], dtype=int64)
```

```python
kmeans.labels_
#Output
#array([1, 3, 1, ..., 0, 1, 0])
```

```python
lut[kmeans.labels_]
#Output
#array([3, 0, 3, ..., 2, 3, 2], dtype=int64)
#```
#idx shows the cluster center labels ordered from lowest to highest consumption level. The appartments for which lut[kmeans.labels_] is 0 / 3 belong to the cluster with the lowest / highest consumption levels.
```


Question:
How to draw a precision-recall curve with interpolation in python? I have drawn a precision-recall curve using sklearn precision_recall_curvefunction and matplotlib package. For those of you who are familiar with precision-recall curve you know that some scientific communities only accept it when its interpolated, similar to this example here. Now my question is if any of you know how to do the interpolation in python? I have been searching for a solution for a while now but with no success! Any help would be greatly appreciated. Solution: Both solutions by @francis and @ali_m are correct and together solved my problem. So, assuming that you get an output from the precision_recall_curve function in sklearn, here is what I did to plot the graph:
```
precision["micro"], recall["micro"], _ = precision_recall_curve(y_test.ravel(),scores.ravel())
pr = copy.deepcopy(precision[0])
rec = copy.deepcopy(recall[0])
prInv = np.fliplr([pr])[0]
recInv = np.fliplr([rec])[0]
j = rec.shape[0]-2
while j>=0:
    if prInv[j+1]>prInv[j]:
        prInv[j]=prInv[j+1]
    j=j-1
decreasing_max_precision = np.maximum.accumulate(prInv[::-1])[::-1]
plt.plot(recInv, decreasing_max_precision, marker= markers[mcounter], label=methodNames[countOfMethods]+': AUC={0:0.2f}'.format(average_precision[0]))
```
And these lines will plot the interpolated curves if you put them in a for loop and pass it the data of each method at each iteration. Note that this will not plot the non-interpolated precision-recall curves.

Response:
@francis's solution can be vectorized using np.maximum.accumulate.
```
import numpy as np
import matplotlib.pyplot as plt

recall = np.linspace(0.0, 1.0, num=42)
precision = np.random.rand(42)*(1.-recall)

# take a running maximum over the reversed vector of precision values, reverse the
# result to match the order of the recall vector
decreasing_max_precision = np.maximum.accumulate(precision[::-1])[::-1]
```
You can also use plt.step to get rid of the for loop used for plotting:
```
fig, ax = plt.subplots(1, 1)
ax.hold(True)
ax.plot(recall, precision, '--b')
ax.step(recall, decreasing_max_precision, '-r')
```


Question:
python - how to append numpy array to a pandas dataframe I have trained a Logistic Regression classifier to predict whether a review is positive or negative. Now, I want to append the predicted probabilities returned by the predict_proba-function to my Pandas data frame containing the reviews. I tried doing something like:
```
test_data['prediction'] = sentiment_model.predict_proba(test_matrix)
```
Obviously, that doesn't work, since predict_proba returns a 2D-numpy array. So, what is the most efficient way of doing this? I created test_matrix with SciKit-Learn's CountVectorizer:
```
vectorizer = CountVectorizer(token_pattern=r'\b\w+\b')
train_matrix = vectorizer.fit_transform(train_data['review_clean'].values.astype('U'))
test_matrix = vectorizer.transform(test_data['review_clean'].values.astype('U'))
```
Sample data looks like:
```
| Review                                     | Prediction         |
| ------------------------------------------ | ------------------ |
| "Toy was great! Our six-year old loved it!"|   0.986            |
```

Response:
Assign the predictions to a variable and then extract the columns from the variable to be assigned to the pandas dataframe cols. If x is the 2D numpy array with predictions,
```
x = sentiment_model.predict_proba(test_matrix)
```
then you can do,
```
test_data['prediction0'] = x[:,0]
test_data['prediction1'] = x[:,1]
```


Question: Normalise between 0 and 1 ignoring NaN For a list of numbers ranging from x to y that may contain NaN, how can I normalise between 0 and 1, ignoring the NaN values (they stay as NaN). Typically I would use MinMaxScaler (ref page) from sklearn.preprocessing, but this cannot handle NaN and recommends imputing the values based on mean or median etc. it doesn't offer the option to ignore all the NaN values.

Response:
consider pd.Series s
```
s = pd.Series(np.random.choice([3, 4, 5, 6, np.nan], 100))
s.hist()
```
Option 1 Min Max Scaling
```
new = s.sub(s.min()).div((s.max() - s.min()))
new.hist()
```
NOT WHAT OP ASKED FOR I put these in because I wanted to Option 2 sigmoid
```
sigmoid = lambda x: 1 / (1 + np.exp(-x))

new = sigmoid(s.sub(s.mean()))
new.hist()
```
Option 3 tanh (hyperbolic tangent)
```
new = np.tanh(s.sub(s.mean())).add(1).div(2)
new.hist()
```


Question:
Get U, Sigma, V* matrix from Truncated SVD in scikit-learn I am using truncated SVD from scikit-learn package. In the definition of SVD, an original matrix A is approxmated as a product A ≈ UΣV* where U and V have orthonormal columns, and Σ is non-negative diagonal. I need to get the U, Σ and V* matrices. Looking at the source code here I found out that V* is stored in self.components_ field after calling fit_transform. Is it possible to get U and Σ matrices? My code:
```
import sklearn.decomposition as skd
import numpy as np

matrix = np.random.random((20,20))
trsvd = skd.TruncatedSVD(n_components=15)
transformed = trsvd.fit_transform(matrix)
VT = trsvd.components_
```

Response:
Looking into the source via the link you provided, TruncatedSVD is basically a wrapper around sklearn.utils.extmath.randomized_svd; you can manually call this yourself like this:
```
from sklearn.utils.extmath import randomized_svd

U, Sigma, VT = randomized_svd(X,
                              n_components=15,
                              n_iter=5,
                              random_state=None)
```


Question: How to output RandomForest Classifier from python? I have trained a RandomForestClassifier from Python Sckit Learn Module with very big dataset, but question is how can I possibly save this model and let other people apply it on their end. Thank you!

Response:
The recommended method is to use joblib, this will result in a much smaller file than a pickle:
```
from sklearn.externals import joblib
joblib.dump(clf, 'filename.pkl')

#then your colleagues can load it

clf = joblib.load('filename.pkl')
```
See the online docs


Question: Stratified Sampling in Pandas I've looked at the Sklearn stratified sampling docs as well as the pandas docs and also Stratified samples from Pandas and sklearn stratified sampling based on a column but they do not address this issue. Im looking for a fast pandas/sklearn/numpy way to generate stratified samples of size n from a dataset. However, for rows with less than the specified sampling number, it should take all of the entries. Concrete example: Thank you! :)

Response:
Use min when passing the number to sample. Consider the dataframe df
```
df = pd.DataFrame(dict(
        A=[1, 1, 1, 2, 2, 2, 2, 3, 4, 4],
        B=range(10)
    ))

df.groupby('A', group_keys=False).apply(lambda x: x.sample(min(len(x), 2)))

   A  B
1  1  1
2  1  2
3  2  3
6  2  6
7  3  7
9  4  9
8  4  8
```


Question: Proximity Matrix in sklearn.ensemble.RandomForestClassifier I'm trying to perform clustering in Python using Random Forests. In the R implementation of Random Forests, there is a flag you can set to get the proximity matrix. I can't seem to find anything similar in the python scikit version of Random Forest. Does anyone know if there is an equivalent calculation for the python version?

Response:
Based on Gilles Louppe answer I have written a function. I don't know if it is effective, but it works. Best regards.
```
def proximityMatrix(model, X, normalize=True):

    terminals = model.apply(X)
    nTrees = terminals.shape[1]

    a = terminals[:,0]
    proxMat = 1*np.equal.outer(a, a)

    for i in range(1, nTrees):
        a = terminals[:,i]
        proxMat += 1*np.equal.outer(a, a)

    if normalize:
        proxMat = proxMat / nTrees

    return proxMat

from sklearn.ensemble import  RandomForestClassifier
from sklearn.datasets import load_breast_cancer
train = load_breast_cancer()

model = RandomForestClassifier(n_estimators=500, max_features=2, min_samples_leaf=40)
model.fit(train.data, train.target)
proximityMatrix(model, train.data, normalize=True)
## array([[ 1.   ,  0.414,  0.77 , ...,  0.146,  0.79 ,  0.002],
##        [ 0.414,  1.   ,  0.362, ...,  0.334,  0.296,  0.008],
##        [ 0.77 ,  0.362,  1.   , ...,  0.218,  0.856,  0.   ],
##        ...,
##        [ 0.146,  0.334,  0.218, ...,  1.   ,  0.21 ,  0.028],
##        [ 0.79 ,  0.296,  0.856, ...,  0.21 ,  1.   ,  0.   ],
##        [ 0.002,  0.008,  0.   , ...,  0.028,  0.   ,  1.   ]])
```


Question:
How to fit a polynomial curve to data using scikit-learn? Problem context Using scikit-learn with Python, I'm trying to fit a quadratic polynomial curve to a set of data, so that the model would be of the form y = a2x^2 + a1x + a0 and the an coefficients will be provided by a model. The problem I don't know how to fit a polynomial curve using that package and there seem to be surprisingly few, clear references on how to do it (I've looked for a while). I've seen this question on doing something similar with NumPy, and also this question which does a more complicated fit than I require. What a good solution would look like Hopefully, a good solution would go around like this (sample adapted from linear fit code that I'm using):
```
x = my_x_data.reshape(len(profile), 1)
y = my_y_data.reshape(len(profile), 1)
regression = linear_model.LinearRegression(degree=2) # or PolynomialRegression(degree=2) or QuadraticRegression()
regression.fit(x, y)
```
I would imagine scikit-learn would have a facility like this, since it's pretty common (for example, in R, the formula for fitting can be provided in-code, and they should be able to be pretty interchangeable for that kind of use-case). The question: What is a good way to do this, or where can I find information about how to do this properly?

Response:
Possible duplicate: https://stats.stackexchange.com/questions/58739/polynomial-regression-using-scikit-learn. Is it crucial for some reason that this be done using scikit-learn? The operation you want can be performed very easily using numpy:
```
z = np.poly1d(np.polyfit(x,y,2))
```
After which z(x) returns the value of the fit at x. A scikit-learn solution would almost certainly be simply a wrapper around the same code.


Question: Why use log-probability estimates in GaussianNB [scikit-learn]? I'm currently using scikit-learn's GaussianNB package. I've noticed that I can choose to return results for the classification several different ways. One way to return a classification is using the predict_log_proba method. Why would I choose to use predict_log_proba versus predict_proba versus predict?

Response: predict just gives you the class for every example predict_proba gives you the probability for every class, and predict is just taking the class which maximal probability predict_log_proba gives you the logarithm of the probabilities, this is often handier as probabilities can become very, very small


Question:
How to get Top 3 or Top N predictions using sklearn's SGDClassifier
```py
from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np
from sklearn import linear_model
arr=['dogs cats lions','apple pineapple orange','water fire earth air', 'sodium potassium calcium']
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(arr)
feature_names = vectorizer.get_feature_names()
Y = ['animals', 'fruits', 'elements','chemicals']
T=["eating apple roasted in fire and enjoying fresh air"]
test = vectorizer.transform(T)
clf = linear_model.SGDClassifier(loss='log')
clf.fit(X,Y)
x=clf.predict(test)
#prints: elements
```
In the above code, clf.predict() prints only 1 best prediction for a sample from list X. I am interested in top 3 predictions for a particular sample in the list X, i know the function predict_proba/predict_log_proba returns a list of all probabilities for each feature in list Y, but it has to sorted and then associated with the features in list Y before getting the top 3 results. Is there any direct and efficient way?

Response:
There is no built-in function, but what is wrong with
```
probs = clf.predict_proba(test)
best_n = np.argsort(probs, axis=1)[-n:]
```
? As suggested by one of the comment, should change [-n:] to [:,-n:]
```
probs = clf.predict_proba(test)
best_n = np.argsort(probs, axis=1)[:,-n:]
```


Question:
How to convert a pandas DataFrame subset of columns AND rows into a numpy array? I'm wondering if there is a simpler, memory efficient way to select a subset of rows and columns from a pandas DataFrame. For instance, given this dataframe:
```
df = DataFrame(np.random.rand(4,5), columns = list('abcde'))
print df

          a         b         c         d         e
0  0.945686  0.000710  0.909158  0.892892  0.326670
1  0.919359  0.667057  0.462478  0.008204  0.473096
2  0.976163  0.621712  0.208423  0.980471  0.048334
3  0.459039  0.788318  0.309892  0.100539  0.753992
```
I want only those rows in which the value for column 'c' is greater than 0.5, but I only need columns 'b' and 'e' for those rows. This is the method that I've come up with - perhaps there is a better "pandas" way?
```
locs = [df.columns.get_loc(_) for _ in ['a', 'd']]
print df[df.c > 0.5][locs]

          a         d
0  0.945686  0.892892
```
My final goal is to convert the result to a numpy array to pass into an sklearn regression algorithm, so I will use the code above like this:
```
training_set = array(df[df.c > 0.5][locs])
```
... and that peeves me since I end up with a huge array copy in memory. Perhaps there's a better way for that too?

Response:
Use its value directly:
```

```python
df[df.c > 0.5][['b', 'e']].values
#Output
#array([[ 0.98836259,  0.82403141],
#       [ 0.337358  ,  0.02054435],
#       [ 0.29271728,  0.37813099],
#       [ 0.70033513,  0.69919695]])
#```
```


Question:
How to normalize a numpy array to a unit vector I would like to convert a NumPy array to a unit vector. More specifically, I am looking for an equivalent version of this normalisation function:
```
def normalize(v):
    norm = np.linalg.norm(v)
    if norm == 0:
       return v
    return v / norm
```
This function handles the situation where vector v has the norm value of 0. Is there any similar functions provided in sklearn or numpy?

Response:
If you're using scikit-learn you can use sklearn.preprocessing.normalize:
```
import numpy as np
from sklearn.preprocessing import normalize

x = np.random.rand(1000)*10
norm1 = x / np.linalg.norm(x)
norm2 = normalize(x[:,np.newaxis], axis=0).ravel()
print np.all(norm1 == norm2)
# True
```