Question:
Scaling / Normalizing pandas column I have a dataframe like:
```
TOTAL | Name
3232     Jane
382      Jack
8291     Jones
```
I'd like to create a newly scaled column in the dataframe called SIZE where SIZE is a number between 5 and 50. For Example:
```
TOTAL | Name | SIZE
3232     Jane   24.413
382      Jack   10
8291     Jones  50
```
I've tried
```
from sklearn.preprocessing import MinMaxScaler
import pandas as pd

scaler=MinMaxScaler(feature_range=(10,50))
df["SIZE"]=scaler.fit_transform(df["TOTAL"])
```
but got Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample. I've tried other things, such as creating a list, transforming it, and appending it back to the dataframe, among other things. What is the easiest way to do this? Thanks!

Response:
Option 1 sklearn You see this problem time and time again, the error really should be indicative of what you need to do. You're basically missing a superfluous dimension on the input. Change df["TOTAL"] to df[["TOTAL"]].
```
df['SIZE'] = scaler.fit_transform(df[["TOTAL"]])
```
```
df
   TOTAL   Name       SIZE
0   3232   Jane  24.413959
1    382   Jack  10.000000
2   8291  Jones  50.000000
```
Option 2 pandas Preferably, I would bypass sklearn and just do the min-max scaling myself.
```
a, b = 10, 50
x, y = df.TOTAL.min(), df.TOTAL.max()
df['SIZE'] = (df.TOTAL - x) / (y - x) * (b - a) + a
```
```
df
   TOTAL   Name       SIZE
0   3232   Jane  24.413959
1    382   Jack  10.000000
2   8291  Jones  50.000000
```
This is essentially what the min-max scaler does, but without the overhead of importing scikit learn (don't do it unless you have to, it's a heavy library).


Question: How to use polars dataframes with scikit-learn? I'm unable to use polars dataframes with scikit-learn for ML training. Currently, I'm preprocessing all dataframes in polars and convert them to pandas for model training in order for it to work. Is there any method to directly use polars dataframes with the scikit-learn API (without converting to pandas first)?

Response:
Since asking the question, scikit-learn 1.4 was released improving compatibility with polars. For example, see the set_output() method of an instance of the sklearn.compose.ColumnTransformer. It can be used as follows. We start with some sample data
```py
import polars as pl

df = pl.DataFrame({
    "num": [1, 2, 3],
    "cat": ["a", "b", "c"],
})
```
and apply the ColumnTransformer as follows.
```py
from sklearn.preprocessing import StandardScaler, OrdinalEncoder
from sklearn.compose import ColumnTransformer

# create column transformer
transformer = ColumnTransformer(
    transformers=[
        ("num", StandardScaler(), ["num"]),
        ("cat", OrdinalEncoder(), ["cat"]),
    ]
)

# enable polars output
transformer.set_output(transform="polars")

# fit and transform polars dataframe
transformer.fit_transform(df)
```
The output again is a pl.DataFrame object.
```
shape: (3, 2)
┌───────────┬──────────┐
│ num__num  ┆ cat__cat │
│ ---       ┆ ---      │
│ f64       ┆ f64      │
╞═══════════╪══════════╡
│ -1.224745 ┆ 0.0      │
│ 0.0       ┆ 1.0      │
│ 1.224745  ┆ 2.0      │
└───────────┴──────────┘
```


Question: KL-Divergence of two GMMs I have two GMMs that I used to fit two different sets of data in the same space, and I would like to calculate the KL-divergence between them. Currently I am using the GMMs defined in sklearn (http://scikit-learn.org/stable/modules/generated/sklearn.mixture.GMM.html) and the SciPy implementation of KL-divergence (http://docs.scipy.org/doc/scipy-dev/reference/generated/scipy.stats.entropy.html) How would I go about doing this? Do I want to just create tons of random points, get their probabilities on each of the two models (call them P and Q) and then use those probabilities as my input? Or is there some more canonical way to do this within the SciPy/SKLearn environment?

Response:
There's no closed form for the KL divergence between GMMs. You can easily do Monte Carlo, though. Recall that KL(p||q) = \int p(x) log(p(x) / q(x)) dx = E_p[ log(p(x) / q(x)). So:
```
def gmm_kl(gmm_p, gmm_q, n_samples=10**5):
    X = gmm_p.sample(n_samples)
    log_p_X, _ = gmm_p.score_samples(X)
    log_q_X, _ = gmm_q.score_samples(X)
    return log_p_X.mean() - log_q_X.mean()
```
(mean(log(p(x) / q(x))) = mean(log(p(x)) - log(q(x))) = mean(log(p(x))) - mean(log(q(x))) is somewhat cheaper computationally.) You don't want to use scipy.stats.entropy; that's for discrete distributions. If you want the symmetrized and smoothed Jensen-Shannon divergence KL(p||(p+q)/2) + KL(q||(p+q)/2) instead, it's pretty similar:
```
def gmm_js(gmm_p, gmm_q, n_samples=10**5):
    X = gmm_p.sample(n_samples)
    log_p_X, _ = gmm_p.score_samples(X)
    log_q_X, _ = gmm_q.score_samples(X)
    log_mix_X = np.logaddexp(log_p_X, log_q_X)

    Y = gmm_q.sample(n_samples)
    log_p_Y, _ = gmm_p.score_samples(Y)
    log_q_Y, _ = gmm_q.score_samples(Y)
    log_mix_Y = np.logaddexp(log_p_Y, log_q_Y)

    return (log_p_X.mean() - (log_mix_X.mean() - np.log(2))
            + log_q_Y.mean() - (log_mix_Y.mean() - np.log(2))) / 2
```
(log_mix_X/log_mix_Y are actually the log of twice the mixture densities; pulling that out of the mean operation saves some flops.)


Question: What does "splitter" attribute in sklearn's DecisionTreeClassifier do? The sklearn DecisionTreeClassifier has a attribute called "splitter" , it is set to "best" by default, what does setting it to "best" or "random" do? I couldn't find enough information from the official documentation.

Response:
There is 2 things to consider, the criterion and the splitter. During all the explaination, I'll use the wine dataset example: Criterion: It is used to evaluate the feature importance. The default one is gini but you can also use entropy. Based on this, the model will define the importance of each feature for the classification. Example: The wine dataset using a "gini" criterion has a feature importance of:
```
alcohol -> 0.04727507393151268
                          malic_acid -> 0.0
                                 ash -> 0.0
                   alcalinity_of_ash -> 0.0
                           magnesium -> 0.0329784450464887
                       total_phenols -> 0.0
                          flavanoids -> 0.1414466773122087
                nonflavanoid_phenols -> 0.0
                     proanthocyanins -> 0.0
                     color_intensity -> 0.0
                                 hue -> 0.08378677906228588
        od280/od315_of_diluted_wines -> 0.3120425747831769
                             proline -> 0.38247044986432716
```
The wine dataset using a "entropy" criterion has a feature importance of:
```
alcohol -> 0.014123729330936566
                          malic_acid -> 0.0
                                 ash -> 0.0
                   alcalinity_of_ash -> 0.02525179137252771
                           magnesium -> 0.0
                       total_phenols -> 0.0
                          flavanoids -> 0.4128453371544815
                nonflavanoid_phenols -> 0.0
                     proanthocyanins -> 0.0
                     color_intensity -> 0.22278576133186542
                                 hue -> 0.011635633063349873
        od280/od315_of_diluted_wines -> 0.0
                             proline -> 0.31335774774683883
```
Results varies with the random_state so I think that only a subset of the dataset is used to compute it. Splitter: The splitter is used to decide which feature and which threshold is used. Using best, the model if taking the feature with the highest importance Using random, the model if taking the feature randomly but with the same distribution (in gini, proline have an importance of 38% so it will be taken in 38% of cases) Example: After training 1000 DecisionTreeClassifier with criterion="gini", splitter="best" and here is the distribution of the "feature number" used at the first split and the 'threshold' It always choses the feature 12 (=proline) with a threshold of 755. This is the head of one of the model trained: By doing the same with splitter= "random", the result is: The threshold is more variant due to the use of different features, here is the result by filtering model having the feature 12 as first split: We can see that the model is also taking randomply the thresholdto split. By looking at the distribution of the feature 12 in regards of classes, we have: The red line being the threshold used when splitter="best". Now, using random, the model will randomly select a threshold value (I think normally distributed with a mean/stdev of the feature but I'm not sure) leading the a distribution centered in the green light and with min max in blue (done with 1353 randomly trained model wtarting with feature 12 for the split) Code to reproduce:
```
from sklearn import datasets
from sklearn.tree import DecisionTreeClassifier, plot_tree, _tree
import numpy as np
import matplotlib.pyplot as plt

wine = datasets.load_wine()

# Feature importance

clf = DecisionTreeClassifier(criterion="gini", splitter='best', random_state=42)
clf = clf.fit(wine.data, wine.target)

for name, val in zip(wine.feature_names, clf.feature_importances_):
    print(f"{name:>40} -> {val}")

print("")
clf = DecisionTreeClassifier(criterion="entropy", splitter='best', random_state=42)
clf = clf.fit(wine.data, wine.target)

for name, val in zip(wine.feature_names, clf.feature_importances_):
    print(f"{name:>40} -> {val}")

# Feature selected first and threshold

features = []
tresholds = []
for random in range(1000):
    clf = DecisionTreeClassifier(criterion="gini", splitter='best', random_state=random)
    clf = clf.fit(wine.data, wine.target)
    features.append(clf.tree_.feature[0])
    tresholds.append(clf.tree_.threshold[0])

# plot distribution
fig, (ax, ax2) = plt.subplots(1, 2, figsize=(20, 5))
ax.hist(features, bins=np.arange(14)-0.5)
ax2.hist(tresholds)
ax.set_title("Number of the first used for split")
ax2.set_title("Value of the threshold")
plt.show()

# plot model
plt.figure(figsize=(20, 12))
plot_tree(clf)
plt.show()

# plot filtered result
threshold_filtered = [val for feat, val in zip(features, tresholds) if feat==12]
fig, ax = plt.subplots(1, 1, figsize=(20, 10))
ax.hist(threshold_filtered)
ax.set_title("Number of the first used for split")
plt.show()

feature_number = 12
X1, X2, X3 = wine.data[wine.target==0][:, feature_number], wine.data[wine.target==1][:, feature_number], wine.data[wine.target==2][:, feature_number]

fig, ax = plt.subplots()
ax.set_title(f'feature {feature_number} - distribution')
ax.boxplot([X1, X2, X3])
ax.hlines(755, 0.5, 3.5, colors="r", linestyles="dashed")
ax.hlines(min(threshold_filtered), 0.5, 3.5, colors="b", linestyles="dashed")
ax.hlines(max(threshold_filtered), 0.5, 3.5, colors="b", linestyles="dashed")
ax.hlines(sum(threshold_filtered)/len(threshold_filtered), 0.5, 3.5, colors="g", linestyles="dashed")
plt.xlabel("Class")
plt.show()
```


Question:
How to print result of clustering in sklearn I have a sparse matrix
```
from scipy.sparse import *
M = csr_matrix((data_np, (rows_np, columns_np)));
```
then I'm doing clustering that way
```
from sklearn.cluster import KMeans
km = KMeans(n_clusters=n, init='random', max_iter=100, n_init=1, verbose=1)
km.fit(M)
```
and my question is extremely noob: how to print the clustering result without any extra information. I don't care about plotting or distances. I just need clustered rows looking that way
```
Cluster 1
row 1
row 2
row 3

Cluster 2
row 4
row 20
row 1000
...
```
How can I get it? Excuse me for this question.

Response:
Time to help myself. After
```
km.fit(M)
```
we run
```
labels = km.predict(M)
```
which returns labels, numpy.ndarray. Number of elements in this array equals number of rows. And each element means that a row belongs to the cluster. For example: if first element is 5 it means that row 1 belongs to cluster 5. Lets put our rows in a dictionary of lists looking this way {cluster_number:[row1, row2, row3], ...}
```
# in row_dict we store actual meanings of rows, in my case it's russian words
clusters = {}
    n = 0
    for item in labels:
        if item in clusters:
            clusters[item].append(row_dict[n])
        else:
            clusters[item] = [row_dict[n]]
        n +=1
```
and print the result
```
for item in clusters:
    print "Cluster ", item
    for i in clusters[item]:
        print i
```


Question:
How to set custom stop words for sklearn CountVectorizer? I'm trying to run LDA (Latent Dirichlet Allocation) on a non-English text dataset. From sklearn's tutorial, there's this part where you count term frequency of the words to feed into the LDA:
```
tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2,
                            max_features=n_features,
                            stop_words='english')
```
Which has built-in stop words feature which is only available for English I think. How could I use my own stop words list for this?

Response:
You may just assign a list of your own words to the stop_words, e.g.:
```
stop_words = (["word1", "word2","word3"])
```


Question:
ValueError: Shape mismatch: if categories is an array, it has to be of shape (n_features,) I have create a simple code to implement OneHotEncoder.
```
from sklearn.preprocessing import OneHotEncoder
X = [[0, 'a'], [0, 'b'], [1, 'a'], [2, 'b']]
onehotencoder = OneHotEncoder(categories=[0])
X = onehotencoder.fit_transform(X).toarray()
```
I just want to use method called fit_transform to the X for index 0, so it means for [0, 0, 1, 2] like what you see in X. But it causes an error like this : ValueError: Shape mismatch: if categories is an array, it has to be of shape (n_features,). Anyone can solve this problem ? I am stuck on it

Response:
You need to use ColumnTransformer to specify the column index not categories parameter. Constructor parameter categories is to tell distinct category values explicitly. E.g. you could provide [0, 1, 2] explicitly, but auto will determine it. Further, you can use slice() object instead.
```
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

X = [[0, 'a'], [0, 'b'], [1, 'a'], [2, 'b']]

ct = ColumnTransformer(
    [('one_hot_encoder', OneHotEncoder(categories='auto'), [0])],   # The column numbers to be transformed (here is [0] but can be [0, 1, 3])
    remainder='passthrough'                                         # Leave the rest of the columns untouched
)

X = ct.fit_transform(X)
```


Question:
How to fit a line using RANSAC in Cartesian coordinates? I am using a 2D Lidar and getting the data as angle and distance with respect to lidar Position. I have to create a floor plan using Lidar and the data is given bellow is represent a room. I want to use the RANSAC algorithm to find the wall of the room. If I could fit RANSAC then I believe somehow I could find the floor flan. I have written a code but it's not fitting into my wall. what modification should I do so that it will fit into my room wall?
```
#(angle, distance)
0,942
0.62,3469
1.25,3350
2.5,3410
3.12,3404
3.75,3403
4.37,3464
5,3441
5.62,3445
6.25,3444
6.87,3455
7.5,3464
8.12,3464
8.75,3477
9.37,3470
10,3504
10.62,3505
11.25,3505
11.87,3516
12.5,3529
13.12,3541
13.75,3543
14.37,3552
15,3559
15.62,3565
16.25,3578
16.87,3591
17.5,3607
18.12,3624
18.75,3634
19.37,3630
20,3651
20.62,3673
21.25,3678
21.87,3697
22.5,3711
23.12,3726
23.75,3744
24.37,3765
25,3780
25.62,3796
26.25,3809
26.87,3830
27.5,3867
28.12,3881
28.75,3854
29.37,3751
30,3689
30.62,3637
31.25,3589
31.87,4029
37.5,3483
50,2734
53.75,1686
54.37,1656
55,1631
55.62,1621
56.25,1608
56.87,1600
57.5,1595
58.12,1598
58.75,1596
59.37,1604
60,1611
60.62,1622
61.25,1644
61.87,1673
65,2212
65.62,2221
66.25,1037
66.87,1299
67.5,2086
67.5,130
83.12,2002
83.75,1995
84.37,1993
85.62,2061
86.25,2043
86.87,2046
87.5,2040
89.37,2082
90,2713
90.62,2245
91.25,2046
93.75,2092
102.5,327
109.37,1349
110,4279
110.62,2177
111.25,2175
111.87,2136
113.12,2151
113.75,2170
114.37,2186
123.12,2066
123.75,2080
124.37,2087
125,2110
125.62,1778
126.25,1732
126.87,428
127.5,1650
128.12,1093
128.75,2206
129.37,2219
130,2243
130.62,2276
131.25,2317
131.87,2319
132.5,2305
133.12,2276
133.75,2253
135,2224
135.62,2202
136.25,2181
136.87,2156
137.5,2131
138.12,2108
138.75,2081
139.37,2068
140,2046
140.62,2028
141.25,1982
141.87,2001
142.5,1985
143.12,2030
152.5,1727
153.12,1728
153.75,1722
154.37,1711
155,1700
155.62,1691
156.25,1683
156.87,1672
157.5,1666
158.12,1655
158.75,1645
159.37,1637
160,1633
160.62,1622
161.25,1621
161.87,1611
162.5,1602
163.12,1597
163.75,1592
164.37,1583
165,1579
165.62,1578
166.25,1571
166.87,1564
167.5,1558
168.12,1552
168.75,1551
169.37,1550
170,1545
170.62,1543
171.25,1540
171.87,1537
172.5,1529
173.12,1527
173.75,1527
174.37,1524
175,1522
175.62,1518
176.25,1519
176.87,1517
177.5,1513
178.12,1510
178.75,1514
180,1514
180.62,1511
181.25,1519
181.87,1513
182.5,1514
183.12,1513
183.75,1513
184.37,1514
185,1518
185.62,1517
186.25,1519
186.87,1517
187.5,1526
188.12,1522
188.75,1526
189.37,1527
190,1530
190.62,1536
191.25,1536
191.87,1541
192.5,1544
193.12,1549
193.75,1553
194.37,1555
195,1553
195.62,1563
196.25,1569
196.87,1570
197.5,1581
198.12,1583
198.75,1591
199.37,1597
200,1601
200.62,1606
202.5,1615
203.12,1626
203.75,1625
204.37,1644
205,1646
205.62,1658
206.25,1663
206.87,1674
207.5,1685
208.12,1703
208.75,1703
209.37,1717
210,1732
210.62,1743
211.25,1750
211.87,1766
212.5,1776
213.12,1791
213.75,1808
214.37,1814
215,1835
215.62,1844
216.25,1854
216.87,1870
217.5,1892
218.12,1909
218.75,1918
219.37,1934
220,1952
220.62,1972
221.87,2023
222.5,2039
223.12,2059
223.75,2082
225,2101
225.62,2122
226.25,2148
226.87,2173
227.5,2190
228.12,2214
228.75,2241
229.37,2275
230,2295
230.62,2324
231.25,2348
231.87,2160
232.5,2416
233.12,2445
233.75,2479
234.37,2520
235.62,2607
236.25,2649
236.87,2156
237.5,2726
238.12,2768
238.75,2806
239.37,2865
240,2912
240.62,2962
241.25,3026
241.87,3078
242.5,3147
243.12,3210
243.75,3276
244.37,3315
245,3307
245.62,3288
246.25,3267
246.87,3253
247.5,3153
248.75,4678
249.37,4563
250,4560
250.62,4504
251.25,4523
251.87,4478
252.5,4452
253.12,4465
253.75,4434
254.37,4421
255,4391
255.62,4371
256.25,4361
256.87,4356
257.5,4348
258.12,4326
258.75,4326
259.37,4331
260,4341
260.62,4270
261.25,4263
261.87,4281
262.5,2992
263.12,2984
263.75,2976
264.37,2983
265,2971
265.62,2963
266.25,2963
266.87,2967
267.5,2968
268.12,2951
268.75,2951
270,2959
270.62,2953
271.25,2500
271.87,5514
272.5,2839
273.12,2706
273.75,2721
274.37,2693
288.12,3010
288.75,2999
289.37,2998
290,3020
290.62,3036
291.25,3083
291.87,3169
292.5,3170
293.12,3196
293.75,3212
294.37,3230
295,3234
295.62,3262
296.25,3273
296.87,3298
297.5,3318
298.12,3333
298.75,3356
299.37,3374
300,3394
300.62,3417
301.25,3427
301.87,3453
302.5,3474
303.12,3490
303.75,3516
304.37,3552
305,3571
305.62,3581
307.5,5224
308.12,5271
308.75,5316
309.37,5411
310,3843
310.62,3892
311.25,3907
311.87,3922
312.5,3985
313.12,4016
313.75,4058
315,4081
315.62,4143
316.25,4190
316.87,4230
317.5,4291
318.12,4353
318.75,4406
319.37,4460
320,4512
320.62,4563
321.25,4507
321.87,4473
322.5,4426
323.12,4398
323.75,4371
324.37,4321
325,4274
325.62,4256
326.25,4215
326.87,4194
327.5,4148
328.12,4104
328.75,4077
329.37,4051
330,4024
330.62,3995
331.25,3971
331.87,3932
332.5,3909
333.12,3898
333.75,3884
334.37,3858
335,3840
335.62,3818
336.25,3791
337.5,3765
338.12,3747
338.75,3720
339.37,3715
340,3689
340.62,3687
341.25,3635
341.87,3632
342.5,3624
343.12,3613
343.75,3613
344.37,3594
345,3595
345.62,3560
346.25,3570
346.87,3543
347.5,3555
348.12,3527
348.75,3512
349.37,3512
350,3521
350.62,3486
351.25,3496
351.87,3477
352.5,3487
353.12,3461
353.75,3460
354.37,3458
355,3453
355.62,3460
356.25,3448
357.5,998
358.12,3442
```
I have converted this data into cartesian and plotted this using Matplotlib. After I have done the RANSAC, it shows a blue line in the picture that is not detecting the wall properly. I should find out 4 walls. I do not know which modification I should do in my code. Here is the code:
```
import numpy as np
from matplotlib import pyplot as plt
import pandas as pd
from sklearn import linear_model, datasets
import math

# scan data is stored in a txt file and getting data from that text file

df = pd.read_csv('scanData.txt',delimiter=',')
angle = df.values[:,0]
distance = df.values[:,1]
cartesian = [(r*math.cos(phi*math.pi/180), r*math.sin(phi*math.pi/180)) for r, phi in zip(distance, angle)]
x, y = map(list, zip(*cartesian))
#print(x)

# coverting this into 2d array
x=  np.array(x)
y=  np.array(y)

x=x.reshape(-1, 1)
y=y.reshape(-1, 1)

lr = linear_model.LinearRegression()
lr.fit(x, y)
ransac = linear_model.RANSACRegressor(max_trials=1000,min_samples=300)
ransac.fit(x, y)

# Predict data of estimated models
line_X = np.arange(x.min(), x.max())[:, np.newaxis]
print(line_X)
line_y = lr.predict(line_X)
line_y_ransac = ransac.predict(line_X)
print(line_y_ransac)
plt.scatter(x,y, color='yellowgreen', marker='.',
            label='Inliers')
plt.plot(line_X, line_y_ransac, color='cornflowerblue', linewidth=1,
         label='RANSAC regressor')
plt.legend(loc='lower right')
plt.xlabel("Input")
plt.ylabel("Response")
plt.show()
```
I will be so glad if you could suggest me to achieve the floor plane and surface area. If you have any questions, please drop it into the comment section.

Response:
I spent some time trying different things and manage with relative ease to get the following result. The thought i had was: Divide points into section. Use RANSAC on each section to get a line estimate. The dividing part was done quite trivialt by comparing distance between incrementing measurements. Not that this is the part that needs to be worked more on, you can see it's flaws on the "yellow part" where two lines are estimated as one section. The result I got was the following (note that chaning parameters will improve the result): This is the code (note that I am not a professional programmer):
```
import matplotlib.pyplot as plt
import numpy as np
from sklearn import linear_model, datasets
from skimage.measure import LineModelND, ransac
import pandas as pd
import math

df = pd.read_csv('scanData.txt',delimiter=',')
angle = df.values[:,0]
distance = df.values[:,1]
cartesian = [(r*math.cos(phi*math.pi/180), r*math.sin(phi*math.pi/180)) for r, phi in zip(distance, angle)]
x, y = map(list, zip(*cartesian))

# coverting this into 2d array
x_data =  np.array(x)
y_data =  np.array(y)

def plot_ransac(segment_data_x, segment_data_y):
    data = np.column_stack([segment_data_x, segment_data_y])

    # fit line using all data
    model = LineModelND()
    model.estimate(data)

    # robustly fit line only using inlier data with RANSAC algorithm
    model_robust, inliers = ransac(data, LineModelND, min_samples=2,
                                   residual_threshold=5, max_trials=1000)
    outliers = inliers == False

    # generate coordinates of estimated models
    line_x = np.array([segment_data_x.min(), segment_data_x.max()])
    line_y = model.predict_y(line_x)
    line_y_robust = model_robust.predict_y(line_x)
    k = (line_y_robust[1] - line_y_robust[0])/(line_x[1]- line_x[0])
    m = line_y_robust[0] - k*line_x[0]
    x0 = (segment_data_y.min() - m)/k
    x1 = (segment_data_y.max() - m)/k
    line_x_y = np.array([x0, x1])
    line_y_robust_y = model_robust.predict_y(line_x_y)
    if (distance(line_x[0], line_y_robust[0], line_x[1], line_y_robust[1]) <
    distance(line_x_y[0], line_y_robust_y[0], line_x_y[1], line_y_robust_y[1])):
        plt.plot(line_x, line_y_robust, '-b', label='Robust line model')
    else:
        plt.plot(line_x_y, line_y_robust_y, '-b', label='Robust line model')

x_segments = []
y_segments = []

def distance(x1,y1,x2,y2):
    return np.sqrt((x1-x2)**2 + (y1-y2)**2)

start = 0
distances = []
for i in range(len(x_data)-1):
    distance_to_point = distance(x_data[i], y_data[i], x_data[i+1], y_data[i+1])
    distances.append(distance_to_point)
    if distance_to_point > 200:
        if i-start>10:
            x_segments.append(x_data[start:i])
            y_segments.append(y_data[start:i])
        start = i+1
    if i == len(x_data)-2:
        if i-start>10:
            x_segments.append(x_data[start:i])
            y_segments.append(y_data[start:i])

plt.plot(x_data, y_data, '.', color = 'grey')
for x_seg, y_seg in zip(x_segments, y_segments):
    plt.plot(x_seg, y_seg,'.', markersize = 10)
    plot_ransac(x_seg, y_seg)
    print('Line is:', distance(x_seg[0], y_seg[0],x_seg[1], y_seg[1]), 'units long')

plt.axis('equal')
plt.show()
```
Hope this is somewhat helpful for you.


Question:
Adding pandas columns to a sparse matrix I have additional derived values for X variables that I want to use in my model.
```
XAll = pd_data[['title','wordcount','sumscores','length']]
y = pd_data['sentiment']
X_train, X_test, y_train, y_test = train_test_split(XAll, y, random_state=1)
```
As I am working with text data in title, I first convert it to a dtm separately:
```
vect = CountVectorizer(max_df=0.5)
vect.fit(X_train['title'])
X_train_dtm = vect.transform(X_train['title'])
column_index = X_train_dtm.indices

print(type(X_train_dtm))    # This is <class 'scipy.sparse.csr.csr_matrix'>
print("X_train_dtm shape",X_train_dtm.get_shape())  # This is (856, 2016)
print("column index:",column_index)     # This is column index: [ 533  754  859 ...,  633  950 1339]
```
Now that I have the text as a document term matrix, I would like to add the other features like 'wordcount','sumscores','length' to X_train_dtm which are numeric. This I shall create the model using the new dtm and thus would be more accurate as I would have inserted additinal features. How do I add additional numeric columns of the pandas dataframe to a sparse csr matrix?

Response:
Found the solution. We can do this using sparse.hstack:
```
from scipy.sparse import hstack
X_train_dtm = hstack((X_train_dtm,np.array(X_train['wordcount'])[:,None]))
```


Question: Balanced Random Forest in scikit-learn (python) I'm wondering if there is an implementation of the Balanced Random Forest (BRF) in recent versions of the scikit-learn package. BRF is used in the case of imbalanced data. It works as normal RF, but for each bootstrapping iteration, it balances the prevalence class by undersampling. For example, given two classes N0 = 100, and N1 = 30 instances, at each random sampling it draws (with replacement) 30 instances from the first class and the same amount of instances from the second class, i.e. it trains a tree on a balanced data set. For more information please refer to this paper. RandomForestClassifier() does have the 'class_weight=' parameter, which might be set to 'balanced', but I'm not sure that it is related to downsampling of the bootsrapped training samples.

Response:
What you're looking for is the BalancedBaggingClassifier from imblearn.
```
imblearn.ensemble.BalancedBaggingClassifier(base_estimator=None,
 n_estimators=10, max_samples=1.0, max_features=1.0, bootstrap=True,
 bootstrap_features=False, oob_score=False, warm_start=False, ratio='auto',
 replacement=False, n_jobs=1, random_state=None, verbose=0)
```
Effectively what it allow you to do is to successively undersample your majority class while fitting an estimator on top. You can use random forest or any base estimator from scikit-learn. Here is an example.


Question:
sklearn kfold returning wrong indexes in python I am using kfold function from sklearn package in python on a df (data frame) with non-contious row indexes. this is the code:
```
kFold = KFold(n_splits=10, shuffle=True, random_state=None)
for train_index, test_index in kFold.split(dfNARemove):...
```
I get some train_index or test_index that doesn't exist in my df. what can I do?

Response:
kFold iterator yields to you positional indices of train and validation objects of DataFrame, not their non-continuous indices. You can access your train and validation objects by using .iloc pandas method:
```
kFold = KFold(n_splits=10, shuffle=True, random_state=None)
for train_index, test_index in kFold.split(dfNARemove):
    train_data = dfNARemove.iloc[train_index]
    test_data = dfNARemove.iloc[test_index]
```
If you want to know, which non-continuous indices used for train_index and test_index on each fold, you can do following:
```
non_continuous_train_index = dfNARemove.index[train_index]
non_continuous_test_index = dfNARemove.index[test_index]
```


Question: Good ROC curve but poor precision-recall curve I have some machine learning results that I don't quite understand. I am using python sciki-learn, with 2+ million data of about 14 features. The classification of 'ab' looks pretty bad on the precision-recall curve, but the ROC for Ab looks just as good as most other groups' classification. What can explain that?

Response: Class imbalance. Unlike the ROC curve, PR curves are very sensitive to imbalance. If you optimize your classifier for good AUC on an unbalanced data you are likely to obtain poor precision-recall results.


Question:
Probability prediction method of KNeighborsClassifier returns only 0 and 1 Can anyone tell me what's the problem with my code? Why I can predict probability of iris dataset by using LinearRegression but, KNeighborsClassifier gives me 0 or 1 while it should give me a result like the one LinearRegression yields?
```
from sklearn.datasets import load_iris
from sklearn import metrics

iris = load_iris()
X = iris.data
y = iris.target

for train_index, test_index in skf:
    X_train, X_test = X_total[train_index], X_total[test_index]
    y_train, y_test = y_total[train_index], y_total[test_index]

from sklearn.linear_model import LogisticRegression
ln = LogisticRegression()
ln.fit(X_train,y_train)

ln.predict_proba(X_test)[:,1]
```
array([ 0.18075722, 0.08906078, 0.14693156, 0.10467766, 0.14823032, 0.70361962, 0.65733216, 0.77864636, 0.67203114, 0.68655163, 0.25219798, 0.3863194 , 0.30735105, 0.13963637, 0.28017798])
```
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=5, algorithm='ball_tree', metric='euclidean')
knn.fit(X_train, y_train)

knn.predict_proba(X_test)[0:10,1]
```
array([ 0., 0., 0., 0., 0., 1., 1., 1., 1., 1.])

Response: Because KNN has very limited concept of probability. Its estimate is simply fraction of votes among nearest neighbours. Increase number of neighbours to 15 or 100 or query point near the decision boundary and you will see more diverse results. Currently your points are simply always having 5 neighbours of the same label (thus probability 0 or 1).


Question:
How to achieve stratified K fold splitting for arbitrary number of categorical variables? I have a dataframe of the form, df:
```
cat_var_1    cat_var_2     num_var_1
0    Orange       Monkey         34
1    Banana        Cat           56
2    Orange        Dog           22
3    Banana       Monkey          6
..
```
Suppose the possible values of cat_var_1 in the dataset have the ratios- ['Orange': 0.6, 'Banana': 0.4] and the possible values of cat_var_2 have the ratios ['Monkey': 0.2, 'Cat': 0.7, 'Dog': 0.1]. How to I split the data into train, test and validation sets (60:20:20 split) such that the ratios of the categorical variables remain preserved? In practice, these variables can be of any number, not just two. Also, clearly, the exact ratios may never be achieved in practice, but we would like it to be as near as possible. I have looked into the StratifiedKFold method from sklearn described here: how to split a dataset into training and validation set keeping ratio between classes? but this is restricted to evaluating on the basis of one categorical variable only. Additionally, I would be grateful if you could provide the complexity of the solution you achieve.

Response:
You can pass df.cat_var_1+ "_" + df.cat_var_2 to argument y of StratifiedShuffleSplit.split(): But here is a method that use DataFrame.groupby:
```
import pandas as pd
import numpy as np

nrows = 10000
p1 = {'Orange': 0.6, 'Banana': 0.4}
p2 = {'Monkey': 0.2, 'Cat': 0.7, 'Dog': 0.1}

c1 = [key for key, val in p1.items() for i in range(int(nrows * val))]
c2 = [key for key, val in p2.items() for i in range(int(nrows * val))]
random.shuffle(c1)
random.shuffle(c2)

df = pd.DataFrame({"c1":c1, "c2":c2, "val":np.random.randint(0, 100, nrows)})

index = []
for key, idx in df.groupby(["c1", "c2"]).groups.items():
    arr = idx.values.copy()
    np.random.shuffle(arr)
    p1 = int(0.6 * len(arr))
    p2 = int(0.8 * len(arr))
    index.append(np.split(arr, [p1, p2]))

idx_train, idx_test, idx_validate = list(map(np.concatenate, zip(*index)))
```


Question:
How can I prevent TfidfVectorizer to get numbers as vocabulary I use TfidfVectorizer like this:
```
from sklearn.feature_extraction.text import TfidfVectorizer
stop_words = stopwords.words("english")
vectorizer = TfidfVectorizer(stop_words=stop_words, min_df=200)
xs['train'] = vectorizer.fit_transform(docs['train'])
xs['test'] = vectorizer.transform(docs['test']).toarray()
```
But when inspecting vectorizer.vocabulary_ I've noticed that it learns pure number features:
```
[(u'00', 0), (u'000', 1), (u'0000', 2), (u'00000', 3), (u'000000', 4)
```
I don't want this. How can I prevent it?

Response:
You could define the token_pattern when initing the vectorizer. The default one is u'(?u)\b\w\w+\b' (the (?u) part is just turning the re.UNICODE flag on). Could fiddle with that until you get what you need. Something like:
```
vectorizer = TfidfVectorizer(stop_words=stop_words,
                             min_df=200,
                             token_pattern=u'(?u)\b\w*[a-zA-Z]\w*\b')
```
Another option (if the fact that numbers appear in your samples matter) is to mask all the numbers before vectorizing.
```
re.sub('\b[0-9][0-9.,-]*\b', 'NUMBER-SPECIAL-TOKEN', sample)
```
This way numbers will hit the same spot in your vectorizer's vocabulary and you won't completely ignore them either.


Question:
How to normalize only certain columns in scikit-learn? I have data similar to the following:
```
[
   [0, 4, 15]
   [0, 3, 7]
   [1, 5, 9]
   [2, 4, 15]
]
```
I used oneHotEncoder http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html#sklearn.preprocessing.OneHotEncoder.fit_transform to preprocess this data so it is suitable for linear regression to give me this:
```
[
   [1, 0, 0, 4, 15]
   [1, 0, 0, 3, 7]
   [0, 1, 0, 5, 9]
   [0, 0, 1, 4, 15]
]
```
However, I then wish to normalise this data. So far I am just normalising the data like so:
```
preprocessing.normalize(data)
```
However, this normalises all the columns including category ones. My questions are the following: How do I normalise only certain columns? Is it desirable to normalise category data, or should I avoid it? Thank you!

Response:
Use numpy to pass a slice of your data to normalize. As for your question about normalizing category data, you will probably get a better answer to that question on CrossValidated. Example for your first question:
```py

```python
import numpy as np
        from sklearn.preprocessing import normalize

        # Values as floats or normalize raises a type error
        X1 = np.array([
                      [1., 0., 0., 4., 15.],
                      [1., 0., 0., 3., 7.],
                      [0., 1., 0., 5., 9.],
                      [0., 0., 1., 4., 15.],
                      ])
```

```python
X1[:, [3,4]] # last two columns
#Output
#array([[  4.,  15.],
#               [  3.,   7.],
#               [  5.,   9.],
#               [  4.,  15.]])
#```
#Normalize the last two columns and assign to a new numpy array, X2.
#```py
```

```python
X2 = normalize(X1[:, [3,4]], axis=0) #axis=0 for column-wise
        X2
#Output
#array([[ 0.49236596,  0.6228411 ],
#               [ 0.36927447,  0.29065918],
#               [ 0.61545745,  0.37370466],
#               [ 0.49236596,  0.6228411 ]])
#```
#Now concatenate X1 and X2 for your desired output.
#```py
```

```python
np.concatenate(( X1[:,[0,1,2]], X2), axis=1)
#Output
#array([[ 1.        ,  0.        ,  0.        ,  0.49236596,  0.6228411 ],
#               [ 1.        ,  0.        ,  0.        ,  0.36927447,  0.29065918],
#               [ 0.        ,  1.        ,  0.        ,  0.61545745,  0.37370466],
#               [ 0.        ,  0.        ,  1.        ,  0.49236596,  0.6228411 ]])
#```
```


Question:
Sklearn Kmeans parameter confusion? So I can run sklearn kmeans as the following:
```
kmeans = KMeans(n_clusters=3,init='random',n_init=10,max_iter=500)
```
But I'm a little confused on what the parameters mean so n_init says: Number of time the k-means algorithm will be run with different centroid seeds. The final results will be the best output of n_init consecutive runs in terms of inertia. and max_iter says: Maximum number of iterations of the k-means algorithm for a single run. But I don't completely understand what that means. Is n_init the number of times the centroids are moved closer to the points mean, given an initial set of centroids? And is max_iter the number of times the whole algorithm is run with new initial centroids? So for example, with max_iter=2,n_init=15, kmeans will choose initial centroids, then move those centroids 15 times and come up with a clustering result. Then kmeans will choose initial centroids again, move those centroids 15 times, and stop. Then, it will pick the best clutering out of the two runs? Thanks for the help! [Edit] Or is the the exact opposite of what I have here... ?

Response: With max_iter=2 and n_init=15, kmeans will choose initial centroids 15 times and move up to twice on each of the 15 runs. The default values are n_init=10 and max_iter=300. This means the initial centroids will be chosen 10 times, and each run will use up to 300 iterations. The best out of those 10 runs will be the final result.


Question:
Scikit-learn χ² (chi-squared) statistic and corresponding contingency table In the docs for the chi-squared univariate feature selection function of scikit-learn http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html, it states This score can be used to select the n_features features with the highest values for the χ² (chi-square) statistic from X, which must contain booleans or frequencies (e.g., term counts in document classification), relative to the classes. I am struggling to understand what the corresponding contingency table would look like, especially in the case of frequency features. For example, consider the below dataset with boolean features and targets:
```
import numpy as np

```python
X = np.random.randint(2, size=50).reshape(10, 5)
#Output
#array([[1, 0, 0, 0, 1],
#       [1, 1, 0, 1, 1],
#       [1, 0, 0, 0, 0],
#       [0, 0, 0, 0, 0],
#       [0, 0, 0, 0, 1],
#       [1, 0, 0, 0, 1],
#       [1, 0, 1, 1, 1],
#       [0, 1, 1, 0, 0],
#       [1, 0, 1, 1, 1],
#       [1, 1, 1, 1, 0]])
```

```python
y = np.random.randint(2, size=10)
#Output
#array([1, 0, 0, 0, 1, 1, 1, 1, 0, 1])
#```
#To construct the contingency table with respect to the first feature, we can do this (excuse my PEP8 violation)
#```
#import scipy as sp
```

```python
contingency_table = sp.sparse.coo_matrix(
   (np.ones_like(y), (X[:, 0], y)),
   shape=(np.unique(X[:, 0]).shape[0], np.unique(y).shape[0])).A
#Output
#array([[1, 2],
#       [3, 4]])
#```
#So now I can calculate the chi-squared statistic and its p-values
#```
```

```python
sp.stats.chi2_contingency(contingency_table)
#Output
#(0.17857142857142855,
# 0.67260381744151676,
# 1,
# array([[ 1.2,  1.8],
#       [ 2.8,  4.2]]))
#```
#And this ought to be consistent with scikit-learn's chi2
#```
#from sklearn.feature_selection import chi2
```

```python
chi2_, pval = chi2(X, y)
```

```python
chi2_[0], pval[0]
#Output
#(0.023809523809523787, 0.87737055606414338)
#```
#...Nope. Have I misinterpreted something? Also, what does the contingency table look like in the case of frequencies? I assumed it would be something like
#```
#contingency_table = sp.sparse.coo_matrix(
#    (np.ones_like(y), (X[:, 0], y)),
#    shape=(X[:, 0].max()+1, np.unique(y).shape[0])).A
#```
#But the corresponding table of expected frequencies will most likely have several zero elements. Edit: To clarify further, consider the first feature X[:, 0] that is, say, gender and the targets y, say, handedness. From this we get the cross tabulation
#```
#Right-handed    Left-handed (!right-handed)
#Male            1               2
#Female (!male)  3               4
#```
#And we can assess the significance of the difference between the two proportions using the Chi-squared test by setting the expected frequency sklearn.feature_selection.chi2 does this directly without resorting to explicitly computing the table and obtains the scores using a more efficient procedure that is equivalent to scipy.stats.chisquare. After explicitly enumerating the table shown above, I wanted to verify it is consistent with chi2 when applying scipy.stats.chi2_contingency and to my dismay, it isn't. I'd like to ask why it isn't.
```

Response:
Given your data,
```

```python
X = array([[1, 0, 0, 0, 1],
       [1, 1, 0, 1, 1],
       [1, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 1],
       [1, 0, 0, 0, 1],
       [1, 0, 1, 1, 1],
       [0, 1, 1, 0, 0],
       [1, 0, 1, 1, 1],
       [1, 1, 1, 1, 0]])
```

```python
y = array([1, 0, 0, 0, 1, 1, 1, 1, 0, 1])
#Output
#```
#this is what feature_selection.chi2 computes:
#```
```

```python
Y = np.vstack([1 - y, y])
```

```python
observed = np.dot(Y, X)
```

```python
observed
#Output
#array([[3, 1, 1, 2, 2],
#       [4, 2, 3, 2, 4]])
#```
#These are the observed feature frequencies, per class, i.e. the contingency table. Then the expected values:
#```
```

```python
feature_count = X.sum(axis=0)
```

```python
class_prob = Y.mean(axis=1)
```

```python
expected = np.dot(feature_count.reshape(-1, 1), class_prob.reshape(1, -1)).T
```

```python
expected
#Output
#array([[ 2.8,  1.2,  1.6,  1.6,  2.4],
#       [ 4.2,  1.8,  2.4,  2.4,  3.6]])
#```
#Finally, it runs a χ² test:
#```
```

```python
from scipy.stats import chisquare
```

```python
score, pval = chisquare(observed, expected)
```

```python
score
#Output
#array([ 0.02380952,  0.05555556,  0.375     ,  0.16666667,  0.11111111])
```

```python
pval
#Output
#array([ 0.87737056,  0.81366372,  0.54029137,  0.6830914 ,  0.73888268])
#```
#The scores are the relevant bit: they're used to sort the features by discriminative power. Note that you get one score and one p-value per feature.
```


Question: Scaling data in scikit-learn SVM While libsvm provides tools for scaling data, with Scikit-Learn (which should be based upon libSVM for the SVC classifier) I find no way to scale my data. Basically I want to use 4 features, of which 3 range from 0 to 1 and the last one is a "big" highly variable number. If I include the fourth feature in libSVM (using the easy.py script which scales my data automatically) I get some very nice results (96% accuracy). If I include the fourth variable in Scikit-Learn the accuracy drops to ~78% - but if I exclude it, I get the same results I get in libSVM when excluding that feature. Therefore I am pretty sure it's a problem of missing scaling. How do I replicate programmatically (i.e. without calling svm-scale) the scaling process of SVM?

Response:
You have that functionality in sklearn.preprocessing:
```

```python
from sklearn import preprocessing
```

```python
X = [[ 1., -1.,  2.],
     [ 2.,  0.,  0.],
     [ 0.,  1., -1.]]
```

```python
X_scaled = preprocessing.scale(X)
```

```python
X_scaled
#Output
#array([[ 0.  ..., -1.22...,  1.33...],
#       [ 1.22...,  0.  ..., -0.26...],
#       [-1.22...,  1.22..., -1.06...]])
#```
#The data will then have zero mean and unit variance.
```


Question: Setting feature weights for KNN I am working with sklearn's implementation of KNN. While my input data has about 20 features, I believe some of the features are more important than others. Is there a way to: set the feature weights for each feature when "training" the KNN learner. learn what the optimal weight values are with or without pre-processing the data. On a related note, I understand generally KNN does not require training but since sklearn implements it using KDTrees, the tree must be generated from the training data. However, this sounds like its turning KNN into a binary tree problem. Is that the case? Thanks.

Response: kNN is simply based on a distance function. When you say "feature two is more important than others" it usually means difference in feature two is worth, say, 10x difference in other coords. Simple way to achive this is by multiplying coord #2 by its weight. So you put into the tree not the original coords but coords multiplied by their respective weights. In case your features are combinations of the coords, you might need to apply appropriate matrix transform on your coords before applying weights, see PCA (principal component analysis). PCA is likely to help you with question 2.


Question:
Keras and Sklearn logreg returning different results I'm comparing the results of a logistic regressor written in Keras to the default Sklearn Logreg. My input is one-dimensional. My output has two classes and I'm interested in the probability that the output belongs to the class 1. I'm expecting the results to be almost identical, but they are not even close. Here is how I generate my random data. Note that X_train, X_test are still vectors, I'm just using capital letters because I'm used to it. Also there is no need for scaling in this case.
```
X = np.linspace(0, 1, 10000)
y = np.random.sample(X.shape)
y = np.where(y<X, 1, 0)
```
Here's cumsum of y plotted over X. Doing a regression here is not rocket science. I do a standard train-test-split:
```
X_train, X_test, y_train, y_test = train_test_split(X, y)
X_train = X_train.reshape(-1,1)
X_test = X_test.reshape(-1,1)
```
Next, I train a default logistic regressor:
```
from sklearn.linear_model import LogisticRegression
sk_lr = LogisticRegression()
sk_lr.fit(X_train, y_train)
sklearn_logreg_result = sk_lr.predict_proba(X_test)[:,1]
```
And a logistic regressor that I write in Keras:
```
from keras.models import Sequential
from keras.layers import Dense
keras_lr = Sequential()
keras_lr.add(Dense(1, activation='sigmoid', input_dim=1))
keras_lr.compile(loss='mse', optimizer='sgd', metrics=['accuracy'])
_ = keras_lr.fit(X_train, y_train, verbose=0)
keras_lr_result = keras_lr.predict(X_test)[:,0]
```
And a hand-made solution:
```
pearson_corr = np.corrcoef(X_train.reshape(X_train.shape[0],), y_train)[0,1]
b = pearson_corr * np.std(y_train) / np.std(X_train)
a = np.mean(y_train) - b * np.mean(X_train)
handmade_result = (a + b * X_test)[:,0]
```
I expect all three to deliver similar results, but here is what happens. This is a reliability diagram using 100 bins. I have played around with loss functions and other parameters, but the Keras logreg stays roughly like this. What might be causing the problem here? edit: Using binary crossentropy is not the solution here, as shown by this plot (note that the input data has changed between the two plots).

Response: While both implementations are a form of Logistic Regression there's quite a few differences. While both solutions converge to a comparable minimum (0.75/0.76 ACC) they are not identical. Optimizer - keras uses vanille SGD where sklearn's LR is based on liblinear which implements trust region Newton method Regularization - sklearn has built in L2 regularization Weights -The weights are randomly initialized and probably sampled from a different distribution.


Question:
ValueError: The number of classes has to be greater than one (python) When passing x,y in fit, I am getting the following error: Traceback (most recent call last): File "C:/Classify/classifier.py", line 95, in train_avg, test_avg, cms = train_model(X, y, "ceps", plot=True) File "C:/Classify/classifier.py", line 47, in train_model clf.fit(X_train, y_train) File "C:\Python27\lib\site-packages\sklearn\svm\base.py", line 676, in fit raise ValueError("The number of classes has to be greater than" ValueError: The number of classes has to be greater than one. Below is my code:
```
def train_model(X, Y, name, plot=False):
"""
    train_model(vector, vector, name[, plot=False])

    Trains and saves model to disk.
"""
labels = np.unique(Y)

cv = ShuffleSplit(n=len(X), n_iter=1, test_size=0.3, indices=True, random_state=0)

train_errors = []
test_errors = []

scores = []
pr_scores = defaultdict(list)
precisions, recalls, thresholds = defaultdict(list), defaultdict(list), defaultdict(list)

roc_scores = defaultdict(list)
tprs = defaultdict(list)
fprs = defaultdict(list)

clfs = []  # for the median

cms = []

for train, test in cv:
    X_train, y_train = X[train], Y[train]
    X_test, y_test = X[test], Y[test]

    clf = LogisticRegression()
    clf.fit(X_train, y_train)
    clfs.append(clf)
```

Response: You probably have only one unique class label in the training set present. As the error messages noted, you need to have at least two unique classes in the dataset. E.g., you can run np.unique(y) to see what the unique class labels in your dataset are.


Question:
classification_report: labels and target_names I have following output from a classification report:
```
precision    recall  f1-score   support

          0     0.6772    0.5214    0.5892       491
          1     0.8688    0.9273    0.8971      1678

avg / total     0.8254    0.8354    0.8274      2169
```
The true labels in the dataset are s and p. Question: How do I know which label is "0" and which is "1"? Or: How can I assign the labels via labels= or target_names= in the correct order?

Response:
It will be arranged in alphabetic order if not specified otherwise. So most probably it will be: 0 -> 'p' 1 -> 's' Anyways, if you pass the actual labels, they should be displayed as it is. For example:
```
y_true = ['p', 's', 'p', 's', 'p']
y_pred = ['p', 'p', 's', 's', 'p']

print(classification_report(y_true, y_pred))

Output:
             precision    recall  f1-score   support

          p       0.67      0.67      0.67         3
          s       0.50      0.50      0.50         2

avg / total       0.60      0.60      0.60         5
```
So no need to do anything. But if you have changed the labels then you can pass them in the target_names param to display in the report. Let's say you have converted 'p' to 0 and 's' to 1, then your code becomes:
```
y_true = [0, 1, 0, 1, 0]
y_pred = [0, 0, 1, 1, 0]

# Without the target_names
print(classification_report(y_true, y_pred))

          0       0.67      0.67      0.67         3
          1       0.50      0.50      0.50         2

avg / total       0.60      0.60      0.60         5

#With target_names
print(classification_report(y_true, y_pred, target_names=['p', 's']))

          p       0.67      0.67      0.67         3
          s       0.50      0.50      0.50         2

avg / total       0.60      0.60      0.60         5
```


Question:
Sklearn's MinMaxScaler only returns zeros I am trying to scale a some number to a range of 0 - 1 using preprocessing from sklearn. Thats what i did:
```
data = [44.645, 44.055, 44.54, 44.04, 43.975, 43.49, 42.04, 42.6, 42.46, 41.405]
min_max_scaler = preprocessing.MinMaxScaler(feature_range=(0, 1))
data_scaled = min_max_scaler.fit_transform([data])
print data_scaled
```
But data_scaled only contains zeros. What am i doing wrong?

Response:
I had the same problem when I tried scaling with MinMaxScaler from sklearn.preprocessing. Scaler returned me zeros when I used a shape a numpy array as list, i.e. [1, n] which looks like the following:
```
data = [[44.645, 44.055, 44.54, 44.04, 43.975, 43.49, 42.04, 42.6, 42.46, 41.405]]
```
I changed the shape of array to [n, 1]. In your case it would like the following
```
data = [[44.645],
        [44.055],
        [44.540],
        [44.040],
        [43.975],
        [43.490],
        [42.040],
        [42.600],
        [42.460],
        [41.405]]
```
Then MinMaxScaler worked in proper way.


Question: What does the KNN algorithm do in the training phase? Unlike other algorithms like linear regressions ,KNN doesn't seems to perform any calculation in the training phase. Like in case of linear regressions it finds the coefficients in the training phase.But what about KNN?

Response: During training phase, KNN arranges the data (sort of indexing process) in order to find the closest neighbors efficiently during the inference phase. Otherwise, it would have to compare each new case during inference with the whole dataset making it quite inefficient. You can read more about it at: https://scikit-learn.org/stable/modules/neighbors.html#nearest-neighbor-algorithms


Question: What does it mean for an attribute name to end in an underscore? Similar question: What's the advantage of a trailing underscore in Python naming?. This addresses advantages/disadvantages, whereas this addresses the reasoning behind doing it, both broadly and specifically to sklearn. I am looking through the sklearn documentation, and I noticed that the sklearn.model_selection.GridSearchCV attributes all end in underscore. For example: cv_results_ best_params_ best_score_ Why is this? What does the underscore do? Please be as broad as possible in your answer (i.e. don't just refer to sklearn's GridSearchCV. I'm assuming this isn't just an sklearn thing, and I have no idea what the appropriate tag is for this so I'm tagging sklearn. Please correct the tags (or me!).

Response:
For sklearn, there is a specific interpretation. Check the sklearn developer guideline, which has a note on this. The convention is used for attributes of estimators that have a meaningful value after fit() was called. These are then used to for instance check if the estimator was fitted, see for instance here:
```
class LinearModel(six.with_metaclass(ABCMeta, BaseEstimator)):
    """Base class for Linear Models"""

    [...]

    def _decision_function(self, X):
        check_is_fitted(self, "coef_")

   [...]
```


Question:
Python scikit-learn SVM Classifier "ValueError: Found array with dim 3. Expected <= 2" I am trying to implement SVM Classifier over MNIST dataset. As my parameters are 3 dimensional its throwing the following error:
```
ValueError: Found array with dim 3. Expected <= 2
```
Following is my code snippet:
```
import mnist
from sklearn import svm

training_images, training_labels = mnist.load_mnist("training", digits = [1,2,3,4])
classifier = svm.SVC()
classifier.fit(training_images, training_labels)
```
Does sklearn support a multi-dimensional classifier?

Response:
One option for fixing the problem would be to reshape the input data into a 2-dimensional array. Let's assume that your training data consists of 10 images which are each represented as an 3x3 matrix and therefore your input data is 3-dimensional.
```
[ [[1,2,3],   [[1,2,3],           [
   [4,5,6],    [4,5,6],            image 10
   [7,8,9]] ,  [7,8,9]]  , ... ,           ] ]
```
We can turn each image into an array of 9 elements in order to convert the dataset into 2-dimensions.
```
dataset_size = len(training_images)
TwoDim_dataset = dataset.reshape(dataset_size,-1)
```
This would turn the data into the following shape:
```
[ [1,2,3,4,5,6,7,8,9]  ,  [1,2,3,4,5,6,7,8,9]  , ... ,  [image 10] ]
```


Question:
CountVectorizer with Pandas dataframe I am using scikit-learn for text processing, but my CountVectorizer isn't giving the output I expect. My CSV file looks like:
```
"Text";"label"
"Here is sentence 1";"label1"
"I am sentence two";"label2"
...
```
and so on. I want to use Bag-of-Words first in order to understand how SVM in python works:
```
import pandas as pd
from sklearn import svm
from sklearn.feature_extraction.text import CountVectorizer

data = pd.read_csv(open('myfile.csv'),sep=';')

target = data["label"]
del data["label"]

# Creating Bag of Words
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(data)
X_train_counts.shape
count_vect.vocabulary_.get(u'algorithm')
```
But when I do print(X_train_counts.shape) I see the output is only (1,1), whereas I have 1048 rows with sentences. What I am doing wrong? I am following this tutorial. (Also the output of count_vect.vocabulary_.get(u'algorithm') is None.)

Response:
The problem is in count_vect.fit_transform(data). The function expects an iterable that yields strings. Unfortunately, these are the wrong strings, which can be verified with a simple example.
```
for x in data:
    print(x)
# Text
```
Only the column names get printed; iterating gives columns instead of the values of data['Text']. You should do this:
```
X_train_counts = count_vect.fit_transform(data.Text)
X_train_counts.shape
# (2, 5)
count_vect.vocabulary_
# {'am': 0, 'here': 1, 'is': 2, 'sentence': 3, 'two': 4}
```


Question:
How do I use sklearn CountVectorizer with both 'word' and 'char' analyzer? - python How do I use sklearn CountVectorizer with both 'word' and 'char' analyzer? http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html I could extract the text features by word or char separately but how do i create a charword_vectorizer? Is there a way to combine the vectorizers? or use more than one analyzer?
```

```python
from sklearn.feature_extraction.text import CountVectorizer
```

```python
word_vectorizer = CountVectorizer(analyzer='word', ngram_range=(1, 2), min_df=1)
```

```python
char_vectorizer = CountVectorizer(analyzer='char', ngram_range=(1, 2), min_df=1)
```

```python
x = ['this is a foo bar', 'you are a foo bar black sheep']
```

```python
word_vectorizer.fit_transform(x)
#Output
#<2x15 sparse matrix of type '<type 'numpy.int64'>'
#    with 18 stored elements in Compressed Sparse Column format>
```

```python
char_vectorizer.fit_transform(x)
#Output
#<2x47 sparse matrix of type '<type 'numpy.int64'>'
#    with 64 stored elements in Compressed Sparse Column format>
```

```python
char_vectorizer.get_feature_names()
#Output
#[u' ', u' a', u' b', u' f', u' i', u' s', u'a', u'a ', u'ac', u'ar', u'b', u'ba', u'bl', u'c', u'ck', u'e', u'e ', u'ee', u'ep', u'f', u'fo', u'h', u'he', u'hi', u'i', u'is', u'k', u'k ', u'l', u'la', u'o', u'o ', u'oo', u'ou', u'p', u'r', u'r ', u're', u's', u's ', u'sh', u't', u'th', u'u', u'u ', u'y', u'yo']
```

```python
word_vectorizer.get_feature_names()
#Output
#[u'are', u'are foo', u'bar', u'bar black', u'black', u'black sheep', u'foo', u'foo bar', u'is', u'is foo', u'sheep', u'this', u'this is', u'you', u'you are']
#```
```

Response:
You can pass a callable as the analyzer argument to get full control over the tokenization, e.g.
```

```python
from pprint import pprint
```

```python
import re
```

```python
x = ['this is a foo bar', 'you are a foo bar black sheep']
```

```python
def words_and_char_bigrams(text):
    words = re.findall(r'\w{3,}', text)
    for w in words:
        yield w
        for i in range(len(w) - 2):
            yield w[i:i+2]

```

```python
v = CountVectorizer(analyzer=words_and_char_bigrams)
```

```python
pprint(v.fit(x).vocabulary_)
#Output
#{'ac': 0,
# 'ar': 1,
# 'are': 2,
# 'ba': 3,
# 'bar': 4,
# 'bl': 5,
# 'black': 6,
# 'ee': 7,
# 'fo': 8,
# 'foo': 9,
# 'he': 10,
# 'hi': 11,
# 'la': 12,
# 'sh': 13,
# 'sheep': 14,
# 'th': 15,
# 'this': 16,
# 'yo': 17,
# 'you': 18}
#```
```


Question:
python sklearn get list of available hyper parameters for model I am using python with sklearn, and would like to get a list of available hyper parameters for a model, how can this be done? Thanks This needs to happen before I initialize the model, when I try to use
```
model.get_params()
```
I get this
```
TypeError: get_params() missing 1 required positional argument: 'self'
```

Response:
This should do it: estimator.get_params() where estimator is the name of your model. To use it on a model you can do the following:
```
reg = RandomForestRegressor()
params = reg.get_params()
# do something...
reg.set_params(params)
reg.fit(X,  y)
```
EDIT: To get the model hyperparameters before you instantiate the class:
```
import inspect
import sklearn

models = [sklearn.ensemble.RandomForestRegressor, sklearn.linear_model.LinearRegression]

for m in models:
    hyperparams = inspect.getargspec(m.__init__).args
    print(hyperparams) # Do something with them here
```
The model hyperparameters are passed in to the constructor in sklearn so we can use the inspect model to see what constructor parameters are available, and thus the hyperparameters. You may need to filter out some arguments that aren't specific to the model such as self and n_jobs.


Question:
ValueError: unknown is not supported in sklearn.RFECV [duplicate] This question already has answers here: ValueError: continuous format is not supported (3 answers) Closed 1 year ago. I was trying to narrow down the number of features really relevant for my classifier using rfecv. This is the code I have written
```
import sklearn
import pandas as p
import numpy as np
import scipy as sp
import pylab as pl
from sklearn import linear_model, cross_validation, metrics
from sklearn.svm import SVC
from sklearn.feature_selection import RFECV
from sklearn.metrics import zero_one_loss
from sklearn import preprocessing
#from sklearn.feature_extraction.text import CountVectorizer
#from sklearn.feature_selection import SelectKBest, chi2

modelType = "notext"

# ----------------------------------------------------------
# Prepare the Data
# ----------------------------------------------------------
training_data = np.array(p.read_table('F:/NYC/NYU/SM/3/SNLP/Project/Data/train.tsv'))
print ("Read Data\n")

# get the target variable and set it as Y so we can predict it
Y = training_data[:,-1]

print(Y)

# not all data is numerical, so we'll have to convert those fields
# fix "is_news":
training_data[:,17] = [0 if x == "?" else 1 for x in training_data[:,17]]

# fix -1 entries in hasDomainLink
training_data[:,14] = [0 if x =="-1" else x for x in training_data[:,10]]

# fix "news_front_page":
training_data[:,20] = [999 if x == "?" else x for x in training_data[:,20]]
training_data[:,20] = [1 if x == "1" else x for x in training_data[:,20]]
training_data[:,20] = [0 if x == "0" else x for x in training_data[:,20]]

# fix "alchemy category":
training_data[:,3] = [0 if x=="arts_entertainment" else x for x in training_data[:,3]]
training_data[:,3] = [1 if x=="business" else x for x in training_data[:,3]]
training_data[:,3] = [2 if x=="computer_internet" else x for x in training_data[:,3]]
training_data[:,3] = [3 if x=="culture_politics" else x for x in training_data[:,3]]
training_data[:,3] = [4 if x=="gaming" else x for x in training_data[:,3]]
training_data[:,3] = [5 if x=="health" else x for x in training_data[:,3]]
training_data[:,3] = [6 if x=="law_crime" else x for x in training_data[:,3]]
training_data[:,3] = [7 if x=="recreation" else x for x in training_data[:,3]]
training_data[:,3] = [8 if x=="religion" else x for x in training_data[:,3]]
training_data[:,3] = [9 if x=="science_technology" else x for x in training_data[:,3]]
training_data[:,3] = [10 if x=="sports" else x for x in training_data[:,3]]
training_data[:,3] = [11 if x=="unknown" else x for x in training_data[:,3]]
training_data[:,3] = [12 if x=="weather" else x for x in training_data[:,3]]
training_data[:,3] = [999 if x=="?" else x for x in training_data[:,3]]

print ("Corrected outliers data\n")

# ----------------------------------------------------------
# Models
# ----------------------------------------------------------
if modelType == "notext":
    print ("no text model\n")
    #ignore features which are useless
    X = training_data[:,list([3, 5, 6, 7, 8, 9, 10, 14, 15, 16, 17, 19, 20, 22, 25])]
    scaler = preprocessing.StandardScaler()
    print("initialized scaler \n")
    scaler.fit(X,Y)
    print("fitted train data and labels\n")
    X = scaler.transform(X)
    print("Transformed train data\n")
    svc = SVC(kernel = "linear")
    print("Initialized SVM\n")
    rfecv = RFECV(estimator = svc, cv = 5, loss_func = zero_one_loss, verbose = 1)
    print("Initialized RFECV\n")
    rfecv.fit(X,Y)
    print("Fitted train data and label\n")
    rfecv.support_
    print ("Optimal Number of features : %d" % rfecv.n_features_)
    savetxt('rfecv.csv', rfecv.ranking_, delimiter=',', fmt='%f')
```
At call of "rfecv.fit(X,Y)" my code throws an error from the metrices.py file "ValueError: unknown is not supported" The error sprouts in sklearn.metrics.metrics:
```
# No metrics support "multiclass-multioutput" format
    if (y_type not in ["binary", "multiclass", "multilabel-indicator", "multilabel-sequences"]):
        raise ValueError("{0} is not supported".format(y_type))
```
This is a classification problem, target values only 0 or 1. The data set can be found at Kaggle Competition Data If anyone can point out where I am going wrong, I would appreciate it.

Response:
RFECV checks target/train data to be of one of types binary, multiclass, multilabel-indicator or multilabel-sequences: 'binary': y contains  1. 'multilabel-indicator': y is a label indicator matrix, an array of two dimensions with at least two columns, and at most 2 unique values. while your Y is unknown, that is 'unknown': y is array-like but none of the above, such as a 3d array, or an array of non-sequence objects. The reason for that is your target data is string (of form "0" and "1") and is loaded with read_table as object:
```

```python
training_data[:, -1].dtype
#Output
#dtype('O')
```

```python
type_of_target(training_data[:, -1])
#Output
#'unknown'
#```
#To solve the issue, you can convert to int:
#```
```

```python
Y = training_data[:, -1].astype(int)
```

```python
type_of_target(Y)
#Output
#'binary'
#```
```


Question: How to get all alpha values of scikit-learn SVM classifier? I need the alpha values, which are the Lagrange multipliers of the SVM dual problem, after training a SVM classifier with scikit-learn. According to the document, it seems that scikit-learn provides only svm.dual_coef_, which is the product of the Lagrange multiplier alpha and the label of a data point. I tried to calculate the alpha value manually by dividing the elements of svm.dual_coef_ by the data label, but since svm.dual_coef_ stores only the coefficients of the support vectors, I'm not sure if I iterate over this array, the order of support vectors would be the same as the order in the original training data. So is there a reliable way to get the alpha values of support vectors?

Response:
As alpha values are by definition positive you can get it through taking abs of dual_coefs:
```
alphas = np.abs(svm.dual_coef_)
```
whis is a direct consequence of the fact that
```
svm.dual_coef_[i] = labels[i] * alphas[i]
```
where labels[i] is either -1 or +1 and alphas[i] are always positive. Futhermore, you can also get each label through
```
labels = np.sign(svm.dual_coef_)
```
using the same observation. This is also why scikit-learn does not store alphas as such - they are uniquely represented by dual_coefs_, together with labels. It is easy to understand it once you analyze all possible cases: labels[i] == -1 and alphas[i] > 0 => dual_coef_[i]  impossible (alphas are non-negative) labels[i] == -1 and alphas[i]== 0 => it is not a support vector labels[i] == +1 and alphas[i] > 0 => dual_coef_[i] > 0 and dual_coef_[i] == alphas[i] == labels[i] * alphas[i] labels[i] == +1 and alphas[i]  impossible (alphas are non-negative) labels[i] == +1 and alphas[i]== 0 => it is not a support vector Consequently, if dual_coef_[i] is positive then it is the alphas[i] coefficient, and it belongs to positive class, and if it is negative, alphas[i] == -dual_coef_[i] and it belongs to negative class.


Question:
How to calculate Cohen's kappa coefficient that measures inter-rater agreement ? ( movie review ) I am using scikit learn and tackle the exercise of predicting movie review rating. I have read on Cohen's kappa ( i Frankly to not understand it fully ), and it's usefulness as a metric of comparison between Observed and Expected accuracy. I have proceeded as usual in applying machine learning algorithm on my corpus, using a bag of words model. I read the Cohen's Kappa is a good way to measure the performance of a classifier. How do i adapt this concept to my prediction problem using sklearn ? Sklearn's documentation is not really explicit on how to proceed on this matter with a document term matrix ( if it's even the right way to do it ) sklearn.metrics.cohen_kappa_score(y1, y2, labels=None, weights=None) this is the example found on the sklearn website:
```
from sklearn.metrics import cohen_kappa_score
y_true = [2, 0, 2, 2, 0, 1]
y_pred = [0, 0, 2, 2, 0, 2]
cohen_kappa_score(y_true, y_pred)
```
Is the Kappa scoring calculation applicable here ? among the people who annotated the reviews in my corpus ? How to write it ? Since all the movie reviews are from different annotators, are they still two annotators to consider in evaluating Cohen's Kappa ? What should I do ? Here's the example I am trying :
```
import pandas as pd
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import StratifiedShuffleSplit

xlsx1 = pd.ExcelFile('App-Music/reviews.xlsx')
'''
review are stored in two columns, one for the review, one for the rating
'''

X = pd.read_excel(xlsx1,'Sheet1').Review
Y = pd.read_excel(xlsx1,'Sheet1').Rating

X_train, X_test, Y_train, Y_test = train_test_split(X_documents, Y, stratify=Y)
new_vect= TfidfVectorizer(ngram_range=(1, 2), stop_words='english')
X_train_dtm = new_vect.fit_transform(X_train.values.astype('U'))
X_test_dtm = new_vect.fit_transform(X_test.values.astype('U'))

new_model.fit(X_train_dtm,Y_train)
new_model.score(X_test_dtm,Y_test)

'''
this is the part where I want to calculate cohen kappa score for comparison
'''
```
I might completely wrong on the idea but i read it in this page concerning sentiment analysis: Ultimately, a tool’s accuracy is merely the percentage of times that human judgment agrees with the tool’s judgment. This degree of agreement among humans is also known as human concordance. There have been various studies run by various people and companies, and they concluded that the rate of human concordance is between 70% and 79%.

Response:
As stated in the documentation of cohen_kappa_score: The kappa statistic is symmetric, so swapping y1 and y2 doesn’t change the value. There is no y_pred, y_true in this metric. The signature as you mentioned in the post is
```
sklearn.metrics.cohen_kappa_score(y1, y2, labels=None, weights=None)
```
There is no thing like the correct and predicted values in this case. Its just the labels by two different persons. So it may have differences because of their perceptions and understanding about the topic. You just need to provide two lists (or arrays) with the labels annotated by different annotators. The ordering doesnt matter. EDIT 1 You said that you have the text reviews. In this case, you need to apply some feature extraction process to identify the labels. This metric is used to find the agreement between two humans labelling the data. Like assigning a class to some data samples. This cannot be used directly on raw text. EDIT 2: Assuming that your y contains only integers (maybe reviews from 1 to 10), this becomes a multiclass classification problem. Its supported by the scikit implementation of cohen_kappa_score. And if I understand correctly the sentiment analysis link you posted, then you should do:
```
Y_pred = new_model.predict(X_test_dtm)
cohen_score = cohen_kappa_score(Y_test, Y_pred)
```


Question:
Combining Grid search and cross validation in scikit learn For improving Support Vector Machine outcomes i have to use grid search for searching better parameters and cross validation. I'm not sure how combining them in scikit-learn. Grid search search best parameters (http://scikit-learn.org/stable/modules/grid_search.html) and cross validation avoid overfitting (http://scikit-learn.org/dev/modules/cross_validation.html)
```
#GRID SEARCH
from sklearn import grid_search
parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
svr = svm.SVC()
clf = grid_search.GridSearchCV(svr, parameters)
#print(clf.fit(X, Y))

#CROSS VALIDATION
from sklearn import cross_validation
X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, Y, test_size=0.4, random_state=0)
clf = svm.SVC(kernel='linear', C=1).fit(X_train, y_train)

print("crossvalidation")
print(clf.score(X_test, y_test))
clf = svm.SVC(kernel='linear', C=1)
scores = cross_validation.cross_val_score(clf, X, Y, cv=3)
print(scores )
```
results:
```
GridSearchCV(cv=None,
   estimator=SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0, degree=3, gamma=0.0,
kernel=rbf, probability=False, shrinking=True, tol=0.001, verbose=False),
   estimator__C=1.0, estimator__cache_size=200,
   estimator__class_weight=None, estimator__coef0=0.0,
   estimator__degree=3, estimator__gamma=0.0, estimator__kernel=rbf,
   estimator__probability=False, estimator__shrinking=True,
   estimator__tol=0.001, estimator__verbose=False, fit_params={},
   iid=True, loss_func=None, n_jobs=1,
   param_grid={'kernel': ('linear', 'rbf'), 'C': [1, 10]},
   pre_dispatch=2*n_jobs, refit=True, score_func=None, verbose=0)

crossvalidation
0.0
[ 0.11111111  0.11111111  0.        ]
```

Response: You should do a development / evaluation split first, run the grid search on the development part and measure a unique final score on the evaluation part at the end: There is an example in the documentation.


Question:
Residual plot for residual vs predicted value in Python I have run a KNN model. Now i want to plot the residual vs predicted value plot. Every example from different websites shows that i have to first run a linear regression model. But i couldn't understand how to do this. Can anyone help? Thanks in advance. Here is my model-
```
train, validate, test = np.split(df.sample(frac=1), [int(.6*len(df)), int(.8*len(df))])
x_train = train.iloc[:,[2,5]].values
y_train = train.iloc[:,4].values
x_validate = validate.iloc[:,[2,5]].values
y_validate = validate.iloc[:,4].values
x_test = test.iloc[:,[2,5]].values
y_test = test.iloc[:,4].values
clf=neighbors.KNeighborsRegressor(n_neighbors = 6)
clf.fit(x_train, y_train)
y_pred = clf.predict(x_validate)
```

Response:
Residuals are nothing but how much your predicted values differ from actual values. So, it's calculated as actual values-predicted values. In your case, it's residuals = y_test-y_pred. Now for the plot, just use this;
```py
import matplotlib.pyplot as plt

plt.scatter(residuals,y_pred)

plt.show()
```


Question:
How to write a custom f1 loss function with weighted average for keras? I am trying to do a multiclass classification in keras. Till now I am using categorical_crossentropy as the loss function. But since the metric required is weighted-f1, I am not sure if categorical_crossentropy is the best loss choice. I was trying to implement a weighted-f1 score in keras using sklearn.metrics.f1_score, but due to the problems in conversion between a tensor and a scalar, I am running into errors. Something like this:
```
def f1_loss(y_true, y_pred):
   return 1 - f1_score(np.argmax(y_true, axis=1), np.argmax(y_pred, axis=1), average='weighted')
```
Followed by
```
model.compile(loss=f1_loss, optimizer=opt)
```
How do I write this loss function in keras? Edit: Shape for y_true and y_pred is (n_samples, n_classes) in my case it is (n_samples, 4) y_true and y_pred both are tensors so sklearn's f1_score cannot work directly on them. I need a function that calculates weighted f1 on tensors.

Response:
The variables are self explained:
```
def f1_weighted(true, pred): #shapes (batch, 4)

    #for metrics include these two lines, for loss, don't include them
    #these are meant to round 'pred' to exactly zeros and ones
    #predLabels = K.argmax(pred, axis=-1)
    #pred = K.one_hot(predLabels, 4)

    ground_positives = K.sum(true, axis=0) + K.epsilon()       # = TP + FN
    pred_positives = K.sum(pred, axis=0) + K.epsilon()         # = TP + FP
    true_positives = K.sum(true * pred, axis=0) + K.epsilon()  # = TP
        #all with shape (4,)

    precision = true_positives / pred_positives
    recall = true_positives / ground_positives
        #both = 1 if ground_positives == 0 or pred_positives == 0
        #shape (4,)

    f1 = 2 * (precision * recall) / (precision + recall + K.epsilon())
        #still with shape (4,)

    weighted_f1 = f1 * ground_positives / K.sum(ground_positives)
    weighted_f1 = K.sum(weighted_f1)

    return 1 - weighted_f1 #for metrics, return only 'weighted_f1'
```
Important notes: This loss will work batchwise (as any Keras loss). So if you are working with small batch sizes, the results will be unstable between each batch, and you may get a bad result. Use big batch sizes, enough to include a significant number of samples for all classes. Since this loss collapses the batch size, you will not be able to use some Keras features that depend on the batch size, such as sample weights, for instance.


Question:
How to use cross_val_score with random_state I get different values for different runs. What am I doing wrong here?
```
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score

X = np.random.random((100,5))
y = np.random.randint(0,2,(100,))
cross_val_score = RandomForestClassifier()
cv = StratifiedKFold(y, random_state=1)
s = cross_val_score(cross_val_score, X, y,scoring='roc_auc', cv=cv)
print(s)
# [ 0.42321429  0.44360902  0.34398496]

s = cross_val_score(cross_val_score, X, y, scoring='roc_auc', cv=cv)
print(s)
# [ 0.42678571  0.46804511  0.36090226]
```

Response:
The mistake you are making is calling the RandomForestClassifier whose default arg, random_state is None. So, it picks up the seed generated by np.random to produce the random output. The random_state in both StratifiedKFold and RandomForestClassifier need to be the same inorder to produce equal arrays of scores of cross validation. Illustration:
```
X=np.random.random((100,5))
y=np.random.randint(0,2,(100,))

clf = RandomForestClassifier(random_state=1)
cv = StratifiedKFold(y, random_state=1)        # Setting random_state is not necessary here
s = cross_val_score(clf, X,y,scoring='roc_auc', cv=cv)
print(s)
##[ 0.57612457  0.29044118  0.30514706]
print(s)
##[ 0.57612457  0.29044118  0.30514706]
```
Another way of countering it would be to not provide random_state args for both RFC and SKF. But, simply providing the np.random.seed(value) to create the random integers at the beginning. These would also create equal arrays at the output.


Question:
Evaluating Logistic regression with cross validation I would like to use cross validation to test/train my dataset and evaluate the performance of the logistic regression model on the entire dataset and not only on the test set (e.g. 25%). These concepts are totally new to me and am not very sure if am doing it right. I would be grateful if anyone could advise me on the right steps to take where I have gone wrong. Part of my code is shown below. Also, how can I plot ROCs for "y2" and "y3" on the same graph with the current one? Thank you
```
import pandas as pd
Data=pd.read_csv ('C:\\Dataset.csv',index_col='SNo')
feature_cols=['A','B','C','D','E']
X=Data[feature_cols]

Y=Data['Status']
Y1=Data['Status1']  # predictions from elsewhere
Y2=Data['Status2'] # predictions from elsewhere

from sklearn.linear_model import LogisticRegression
logreg=LogisticRegression()
logreg.fit(X_train,y_train)

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

from sklearn import metrics, cross_validation
predicted = cross_validation.cross_val_predict(logreg, X, y, cv=10)
metrics.accuracy_score(y, predicted)

from sklearn.cross_validation import cross_val_score
accuracy = cross_val_score(logreg, X, y, cv=10,scoring='accuracy')
print (accuracy)
print (cross_val_score(logreg, X, y, cv=10,scoring='accuracy').mean())

from nltk import ConfusionMatrix
print (ConfusionMatrix(list(y), list(predicted)))
#print (ConfusionMatrix(list(y), list(yexpert)))

# sensitivity:
print (metrics.recall_score(y, predicted) )

import matplotlib.pyplot as plt
probs = logreg.predict_proba(X)[:, 1]
plt.hist(probs)
plt.show()

# use 0.5 cutoff for predicting 'default'
import numpy as np
preds = np.where(probs > 0.5, 1, 0)
print (ConfusionMatrix(list(y), list(preds)))

# check accuracy, sensitivity, specificity
print (metrics.accuracy_score(y, predicted))

#ROC CURVES and AUC
# plot ROC curve
fpr, tpr, thresholds = metrics.roc_curve(y, probs)
plt.plot(fpr, tpr)
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate)')
plt.show()

# calculate AUC
print (metrics.roc_auc_score(y, probs))

# use AUC as evaluation metric for cross-validation
from sklearn.cross_validation import cross_val_score
logreg = LogisticRegression()
cross_val_score(logreg, X, y, cv=10, scoring='roc_auc').mean()
```

Response:
You got it almost right. cross_validation.cross_val_predict gives you predictions for the entire dataset. You just need to remove logreg.fit earlier in the code. Specifically, what it does is the following: It divides your dataset in to n folds and in each iteration it leaves one of the folds out as the test set and trains the model on the rest of the folds (n-1 folds). So, in the end you will get predictions for the entire data. Let's illustrate this with one of the built-in datasets in sklearn, iris. This dataset contains 150 training samples with 4 features. iris['data'] is X and iris['target'] is y
```

```python
iris['data'].shape
#Output
#(150, 4)
#```
#To get predictions on the entire set with cross validation you can do the following:
#```
#from sklearn.linear_model import LogisticRegression
#from sklearn import metrics, cross_validation
#from sklearn import datasets
#iris = datasets.load_iris()
#predicted = cross_validation.cross_val_predict(LogisticRegression(), iris['data'], iris['target'], cv=10)
#print metrics.accuracy_score(iris['target'], predicted)
#
#Out [1] : 0.9537
#
#print metrics.classification_report(iris['target'], predicted)
#
#Out [2] :
#                     precision    recall  f1-score   support
#
#                0       1.00      1.00      1.00        50
#                1       0.96      0.90      0.93        50
#                2       0.91      0.96      0.93        50
#
#      avg / total       0.95      0.95      0.95       150
#```
#So, back to your code. All you need is this:
#```
#from sklearn import metrics, cross_validation
#logreg=LogisticRegression()
#predicted = cross_validation.cross_val_predict(logreg, X, y, cv=10)
#print metrics.accuracy_score(y, predicted)
#print metrics.classification_report(y, predicted)
#```
#For plotting ROC in multi-class classification, you can follow this tutorial which gives you something like the following: In general, sklearn has very good tutorials and documentation. I strongly recommend reading their tutorial on cross_validation.
```


Question: Sklearn Model (Python) with NodeJS (Express): how to connect both? I have a web server using NodeJS - Express and I have a Scikit-Learn (machine learning) model pickled (dumped) in the same machine. What I need is to demonstrate the model by sending/receiving data from it to the server. I want to load the model on startup of the web server and keep "listening" for data inputs. When receive data, executes a prediction and send it back. I am relatively new to Python. From what I've seen I could use a "Child Process" to execute that. I also saw some modules that run Python script from Node. The problem is I want to load the model once and let it be for as long as the server is on. I don't want to keep loading the model every time due to it's size. How is the best way to perform that? The idea is running everything in a AWS machine. Thank you in advance.

Response: My recommendation: write a simple python web service (personally recommend flask) and deploy your ML model. Then you can easily send requests to your python web service from your node back-end. You wouldn't have a problem with the initial model loading. it is done once in the app startup, and then you're good to go DO NOT GO FOR SCRIPT EXECUTIONS AND CHILD PROCESSES!!! I just wrote it in bold-italic all caps so to be sure you wouldn't do that. Believe me... it potentially go very very south, with all that zombie processes upon job termination and other stuff. let's just simply say it's not the standard way to do that. You need to think about multi-request handling. I think flask now has it by default I am just giving you general hints because your problem has been generally introduced.


Question:
predict_proba or decision_function as estimator "confidence" I'm using LogisticRegression as a model to train an estimator in scikit-learn. The features I use are (mostly) categorical; and so are the labels. Therefore, I use a DictVectorizer and a LabelEncoder, respectively, to encode the values properly. The training part is fairly straightforward, but I'm having problems with the test part. The simple thing to do is to use the "predict" method of the trained model and get the predicted label. However, for the processing I need to do afterwards, I need the probability for each possible label (class) for each particular instance. I decided to use the "predict_proba" method. However, I get different results for the same test instance, whether I use this method when the instance is by itself or accompanied by others. Next, is a code that reproduces the problem.
```
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction import DictVectorizer
from sklearn.preprocessing import LabelEncoder

X_real = [{'head': u'n\xe3o', 'dep_rel': u'ADVL'},
          {'head': u'v\xe3o', 'dep_rel': u'ACC'},
          {'head': u'empresa', 'dep_rel': u'SUBJ'},
          {'head': u'era', 'dep_rel': u'ACC'},
          {'head': u't\xeam', 'dep_rel': u'ACC'},
          {'head': u'import\xe2ncia', 'dep_rel': u'PIV'},
          {'head': u'balan\xe7o', 'dep_rel': u'SUBJ'},
          {'head': u'ocupam', 'dep_rel': u'ACC'},
          {'head': u'acesso', 'dep_rel': u'PRED'},
          {'head': u'elas', 'dep_rel': u'SUBJ'},
          {'head': u'assinaram', 'dep_rel': u'ACC'},
          {'head': u'agredido', 'dep_rel': u'SUBJ'},
          {'head': u'pol\xedcia', 'dep_rel': u'ADVL'},
          {'head': u'se', 'dep_rel': u'ACC'}]
y_real = [u'AM-NEG', u'A1', u'A0', u'A1', u'A1', u'A1', u'A0', u'A1', u'AM-ADV', u'A0', u'A1', u'A0', u'A2', u'A1']

feat_encoder =  DictVectorizer()
feat_encoder.fit(X_real)

label_encoder = LabelEncoder()
label_encoder.fit(y_real)

model = LogisticRegression()
model.fit(feat_encoder.transform(X_real), label_encoder.transform(y_real))

print "Test 1..."
X_test1 = [{'head': u'governo', 'dep_rel': u'SUBJ'}]
X_test1_encoded = feat_encoder.transform(X_test1)
print "Features Encoded"
print X_test1_encoded
print "Shape"
print X_test1_encoded.shape
print "decision_function:"
print model.decision_function(X_test1_encoded)
print "predict_proba:"
print model.predict_proba(X_test1_encoded)

print "Test 2..."
X_test2 = [{'head': u'governo', 'dep_rel': u'SUBJ'},
           {'head': u'atrav\xe9s', 'dep_rel': u'ADVL'},
           {'head': u'configuram', 'dep_rel': u'ACC'}]

X_test2_encoded = feat_encoder.transform(X_test2)
print "Features Encoded"
print X_test2_encoded
print "Shape"
print X_test2_encoded.shape
print "decision_function:"
print model.decision_function(X_test2_encoded)
print "predict_proba:"
print model.predict_proba(X_test2_encoded)

print "Test 3..."
X_test3 = [{'head': u'governo', 'dep_rel': u'SUBJ'},
           {'head': u'atrav\xe9s', 'dep_rel': u'ADVL'},
           {'head': u'configuram', 'dep_rel': u'ACC'},
           {'head': u'configuram', 'dep_rel': u'ACC'},]

X_test3_encoded = feat_encoder.transform(X_test3)
print "Features Encoded"
print X_test3_encoded
print "Shape"
print X_test3_encoded.shape
print "decision_function:"
print model.decision_function(X_test3_encoded)
print "predict_proba:"
print model.predict_proba(X_test3_encoded)
```
Following is the output obtained:
```
Test 1...
Features Encoded
  (0, 4)    1.0
Shape
(1, 19)
decision_function:
[[ 0.55372615 -1.02949707 -1.75474347 -1.73324726 -1.75474347]]
predict_proba:
[[ 1.  1.  1.  1.  1.]]
Test 2...
Features Encoded
  (0, 4)    1.0
  (1, 1)    1.0
  (2, 0)    1.0
Shape
(3, 19)
decision_function:
[[ 0.55372615 -1.02949707 -1.75474347 -1.73324726 -1.75474347]
 [-1.07370197 -0.69103629 -0.89306092 -1.51402163 -0.89306092]
 [-1.55921001  1.11775556 -1.92080112 -1.90133404 -1.92080112]]
predict_proba:
[[ 0.59710757  0.19486904  0.26065002  0.32612646  0.26065002]
 [ 0.23950111  0.24715931  0.51348452  0.3916478   0.51348452]
 [ 0.16339132  0.55797165  0.22586546  0.28222574  0.22586546]]
Test 3...
Features Encoded
  (0, 4)    1.0
  (1, 1)    1.0
  (2, 0)    1.0
  (3, 0)    1.0
Shape
(4, 19)
decision_function:
[[ 0.55372615 -1.02949707 -1.75474347 -1.73324726 -1.75474347]
 [-1.07370197 -0.69103629 -0.89306092 -1.51402163 -0.89306092]
 [-1.55921001  1.11775556 -1.92080112 -1.90133404 -1.92080112]
 [-1.55921001  1.11775556 -1.92080112 -1.90133404 -1.92080112]]
predict_proba:
[[ 0.5132474   0.12507868  0.21262531  0.25434403  0.21262531]
 [ 0.20586462  0.15864173  0.4188751   0.30544372  0.4188751 ]
 [ 0.14044399  0.3581398   0.1842498   0.22010613  0.1842498 ]
 [ 0.14044399  0.3581398   0.1842498   0.22010613  0.1842498 ]]
```
As can be seen, the values obtained with "predict_proba" for the instance in "X_test1" change when that same instance is with others in X_test2. Also, "X_test3" just reproduces the "X_test2" and adds one more instance (that is equal to the last in "X_test2"), but the probability values for all of them change. Why does this happen? Also, I find it really strange that ALL the probabilities for "X_test1" are 1, shouldn't the sum of all be 1? Now, if instead of using "predict_proba" I use "decision_function", I get the consistency in the values obtained that I need. The problem is that I get negative coefficients, and even some of the positives ones are greater than 1. So, what should I use? Why do the values of "predict_proba" change that way? Am I not understanding correctly what those values mean? Thanks in advance for any help you could give me. UPDATE As suggested, I changed the code so as to also print the encoded "X_test1", "X_test2" and "X_test3", as well as their shapes. This doesn't appear to be the problem, as the encoding is consistant for the same instances between the test sets.

Response: As indicated on the question's comments, the error was caused by a bug in the implementation for the version of scikit-learn I was using. The problem was solved updating to the most recent stable version 0.12.1


Question:
HOW TO LABEL the FEATURE IMPORTANCE with forests of trees? I use sklearn to plot the feature importance for forests of trees. The dataframe is named 'heart'. Here the code to extract the list of the sorted features:
```
importances = extc.feature_importances_
indices = np.argsort(importances)[::-1]
print("Feature ranking:")

for f in range(heart_train.shape[1]):
    print("%d. feature %d (%f)" % (f + 1, indices[f], importances[indices[f]]))
```
Then I plot the list in this way:
```
f, ax = plt.subplots(figsize=(11, 9))
plt.title("Feature ranking", fontsize = 20)
plt.bar(range(heart_train.shape[1]), importances[indices],
    color="b",
    align="center")
plt.xticks(range(heart_train.shape[1]), indices)
plt.xlim([-1, heart_train.shape[1]])
plt.ylabel("importance", fontsize = 18)
plt.xlabel("index of the feature", fontsize = 18)
```
and I get a plot like this: My question is: how could I substitute the NUMBER of the feature with the NAME of the feature in order to turn the plot more understandable? I tried to convert the string containing the name of the feature (which is the name of each column of the data frame), but I cannot reach my goal. Thanks

Response:
The problem is here:
```
plt.xticks(range(heart_train.shape[1]), indices)
```
indices is an array of indices returned from your np.argsort(importances)[::-1], it doesn't have the feature names you want to appear as ticks on your X axis. You need something like this, assuming df is your Pandas DataFrame
```
feature_names = df.columns # e.g. ['A', 'B', 'C', 'D', 'E']
plt.xticks(range(heart_train.shape[1]), feature_names)
```


Question:
Receiving KeyError: "None of [Int64Index([ ... dtype='int64', length=1323)] are in the [columns]" SUMMARY When feeding test and train data into a ROC curve plot, I receive the following error: KeyError: "None of [Int64Index([ 0, 1, 2, ... dtype='int64', length=1323)] are in the [columns]" The error seems to be saying that it doesn't like the format of my data, but it worked when run the first time and I haven't been able to get it to run again. Am I incorrectly splitting my data or sending incorrectly formatted data into my function? WHAT I'VE TRIED Read through several StackOverflow posts with the same KeyError Re-ead through scikit-learn example I followed Reviewed previous versions of my code to troubleshoot I am running this within a CoLab document and it can be viewed here CODE I am using standard dataframes to pull in my X and Y sets:
```
X = df_full.drop(['Attrition'], axis=1)
y = df_full['Attrition'].as_matrix()
```
The KeyError traces back to the 8th line here:
```
def roc_plot(X, Y, Model):
    tprs = []
    aucs = []
    mean_fpr = np.linspace(0, 1, 100)
    plt.figure(figsize=(12,8))
    i = 0
    for train, test in kf.split(X, Y):
        probas_ = model.fit(X[train], Y[train]).predict_proba(X[test])
        # Compute ROC curve and area the curve
        fpr, tpr, thresholds = roc_curve(Y[test], probas_[:, 1])
        tprs.append(np.interp(mean_fpr, fpr, tpr))
        tprs[-1][0] = 0.0
        roc_auc = auc(fpr, tpr)
        aucs.append(roc_auc)
        plt.plot(fpr, tpr, lw=1, alpha=0.3,
                 label='ROC fold %d (AUC = %0.2f)' % (i, roc_auc))

        i += 1
    plt.plot([0, 1], [0, 1], linestyle='--', lw=2, color='r',
             label='Chance', alpha=.8)

    mean_tpr = np.mean(tprs, axis=0)
    mean_tpr[-1] = 1.0
    mean_auc = auc(mean_fpr, mean_tpr)
    std_auc = np.std(aucs)
    plt.plot(mean_fpr, mean_tpr, color='b',
             label=r'Mean ROC (AUC = %0.2f $\pm$ %0.2f)' % (mean_auc, std_auc),
             lw=2, alpha=.8)

    std_tpr = np.std(tprs, axis=0)
    tprs_upper = np.minimum(mean_tpr + std_tpr, 1)
    tprs_lower = np.maximum(mean_tpr - std_tpr, 0)
    plt.fill_between(mean_fpr, tprs_lower, tprs_upper, color='grey', alpha=.2,
                     label=r'$\pm$ 1 std. dev.')

    plt.xlim([-0.05, 1.05])
    plt.ylim([-0.05, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver operating characteristic example')
    plt.legend(loc="lower right")
    plt.show()
```
It happens when I run the following with the function:
```
model = XGBClassifier() # Create the Model
roc_plot(X, Y, Model)
```
EXPECTED RESULT I should be able to feed the data, X and Y, into my function.

Response:
in this piece of code train, test are arrays of indices, while you using it as a columns when selection from DataFrame:
```
for train, test in kf.split(X, Y):
    probas_ = model.fit(X[train], Y[train]).predict_proba(X[test])
```
you should use iloc instead:
```
probas_ = model.fit(X.iloc[train], Y.iloc[train]).predict_proba(X.iloc[test])
```