Question:
Object of type 'ndarray' is not JSON serializable I am new to python and machine learning. I have a Linear Regression model which is able to predict output based on the input which I have dumped to be used with a web service. See the code below:
```
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)

        regression_model = LinearRegression()
        regression_model.fit(X_train, y_train)
    print(regression_model.predict(np.array([[21, 0, 0, 0, 1, 0, 0, 1, 1, 1]]))) # this is returning my expected output

joblib.dump(regression_model, '../trainedModels/MyTrainedModel.pkl')
```
Using flask I am trying this to expose as a web service as below:
```
@app.route('/predict', methods=['POST'])
def predict():

    X = [[21, 0, 0, 0, 1, 0, 0, 1, 1, 1]]
    model = joblib.load('../trainedModels/MyTrainedModel.pkl')
    prediction = model.predict(np.array(X).tolist())
    return jsonify({'prediction': list(prediction)})
```
But it is throwing the following exception
```
Object of type 'ndarray' is not JSON serializable
```
I tried NumPy array is not JSON serializable but still the same error. How can i solve this issue

Response:
Try to convert your ndarray with tolist() method:
```
prediction = model.predict(np.array(X).tolist()).tolist()
return jsonify({'prediction': prediction})
```
Example with json package:
```
a = np.array([1,2,3,4,5]).tolist()
json.dumps({"prediction": a})
```
That should output:
```
'{"prediction": [1, 2, 3, 4, 5]}'
```


Question: PCA inverse transform manually I am using scikit-learn. The nature of my application is such that I do the fitting offline, and then can only use the resulting coefficients online(on the fly), to manually calculate various objectives. The transform is simple, it is just data * pca.components_, i.e. simple dot product. However, I have no idea how to perform the inverse transform. Which field of the pca object contains the relevant coefficients for the inverse transform? How do I calculate the inverse transform? Specifically, I am referring to the PCA.inverse_transform() method call available in the sklearn.decomposition.PCA package: how can I manually reproduce its functionality using various coefficients calculated by the PCA?

Response:
1) transform is not data * pca.components_. Firstly, * is not dot product for numpy array. It is element-wise multiplication. To perform dot product, you need to use np.dot. Secondly, the shape of PCA.components_ is (n_components, n_features) while the shape of data to transform is (n_samples, n_features), so you need to transpose PCA.components_ to perform dot product. Moreover, the first step of transform is to subtract the mean, therefore if you do it manually, you also need to subtract the mean at first. The correct way to transform is
```
data_reduced = np.dot(data - pca.mean_, pca.components_.T)
```
2) inverse_transform is just the inverse process of transform
```
data_original = np.dot(data_reduced, pca.components_) + pca.mean_
```
If your data already has zero mean in each column, you can ignore the pca.mean_ above, for example
```
import numpy as np
from sklearn.decomposition import PCA

pca = PCA(n_components=3)
pca.fit(data)

data_reduced = np.dot(data, pca.components_.T) # transform
data_original = np.dot(data_reduced, pca.components_) # inverse_transform
```


Question:
Python RandomForest - Unknown label Error I have trouble using RandomForest fit function This is my training set
```
P1      Tp1           IrrPOA     Gz          Drz2
0        0.0     7.7           0.0       -1.4        -0.3
1        0.0     7.7           0.0       -1.4        -0.3
2        ...     ...           ...        ...         ...
3        49.4    7.5           0.0       -1.4        -0.3
4        47.4    7.5           0.0       -1.4        -0.3
... (10k rows)
```
I want to predict P1 thanks to all the other variables using sklearn.ensemble RandomForest
```
colsRes = ['P1']
X_train = train.drop(colsRes, axis = 1)
Y_train = pd.DataFrame(train[colsRes])
rf = RandomForestClassifier(n_estimators=100)
rf.fit(X_train, Y_train)
```
Here is the error I get:
```
ValueError: Unknown label type: array([[  0. ],
       [  0. ],
       [  0. ],
       ...,
       [ 49.4],
       [ 47.4],
```
I did not find anything about this label error, I use Python 3.5. Any advice would be a great help !

Response:
When you are passing label (y) data to rf.fit(X,y), it expects y to be 1D list. Slicing the Panda frame always result in a 2D list. So, conflict raised in your use-case. You need to convert the 2D list provided by pandas DataFrame to a 1D list as expected by fit function. Try using 1D list first:
```
Y_train = list(train.P1.values)
```
If this does not solve the problem, you can try with solution mentioned in MultinomialNB error: "Unknown Label Type":
```
Y_train = np.asarray(train['P1'], dtype="|S6")
```
So your code becomes,
```
colsRes = ['P1']
X_train = train.drop(colsRes, axis = 1)
Y_train = np.asarray(train['P1'], dtype="|S6")
rf = RandomForestClassifier(n_estimators=100)
rf.fit(X_train, Y_train)
```


Question:
Significant mismatch between `r2_score` of `scikit-learn` and the R^2 calculation Question Why is there a significant difference between the r2_score function in scikit-learn and the formula for the Coefficient of Determination as described in Wikipedia? Which is the correct one? Context I'm using with Python 3.5 to predict linear and quadratic models, and one of the measures of goodness of fit that I'm trying out is the . However, while testing, there's a marked difference between the r2_score metric in scikit-learn and the calculation provided in Wikipedia. Code I'm providing my code here as reference, which computes the example in the Wikipedia page linked above.
```
from sklearn.metrics import r2_score
import numpy

y = [1, 2, 3, 4, 5]
f = [1.9, 3.7, 5.8, 8.0, 9.6]

# Convert to numpy array and ensure double precision to avoid single precision errors
observed = numpy.array(y, dtype=numpy.float64)
predicted = numpy.array(f, dtype=numpy.float64)

scipy_value = r2_score(observed, predicted)

```python
scipy_value:
#Output
#```
#As is evident, the scipy calculated value is -3.8699999999999992while the reference value in Wikipedia is 0.998. Thank you! UPDATE: This is different from this question about how R^2 is calculated in scikit-learn as what I'm trying to understand and have clarified is the discrepancy between both results. That question states that the formula used in scikit is the same as Wikipedia's which should not result in different values. UPDATE #2: It turns out I made a mistake reading the Wikipedia article's example. Answers and comments below mention that the example I provide is for the linear, least squares fit of the (x, y) values in the example. For that, the answer in Wikipedia's article is correct. For that, the R^2 calue provided is 0.998. For the R^2 between both vectors, scikit's answer is also correct. Thanks a lot for your help!
```

Response:
I think you have misinterpreted wikipedia. The example on wikipedia does not state:
```
y = [1, 2, 3, 4, 5]
f = [1.9, 3.7, 5.8, 8.0, 9.6]
R^2 = 0.998
```
Instead, it says that the R^2 for a linear least-squares fit to the data:
```
x = [1, 2, 3, 4, 5]
y = [1.9, 3.7, 5.8, 8.0, 9.6]
```
is equal to 0.998 Consider this script, which first uses np.linalg.lstsq to find the least squares fit, and the uses both methods to find an R^2 of 0.998 for both:
```
import numpy as np
from sklearn.metrics import r2_score

x = np.arange(1, 6, 1)
y = np.array([1.9, 3.7, 5.8, 8.0, 9.6])

A = np.vstack([x, np.ones(len(x))]).T

# Use numpy's least squares function
m, c = np.linalg.lstsq(A, y)[0]

print(m, c)
# 1.97 -0.11

# Define the values of our least squares fit
f = m * x + c

print(f)
# [ 1.86  3.83  5.8   7.77  9.74]

# Calculate R^2 explicitly
yminusf2 = (y - f)**2
sserr = sum(yminusf2)
mean = float(sum(y)) / float(len(y))
yminusmean2 = (y - mean)**2
sstot = sum(yminusmean2)
R2 = 1. -(sserr / sstot)

print(R2)
# 0.99766066838

# Use scikit
print(r2_score(y,f))
# 0.99766066838

r2_score(y,f) == R2
# True
```


Question: How can I pass a preprocessor to TfidfVectorizer? - sklearn - python How can I pass a preprocessor to TfidfVectorizer? I made a function that takes a string and returns a preprocessed string then I set processor parameter to that function "preprocessor=preprocess", but it doesn't work. I've searched so many times, but I didn't found any example as if no one use it. I have another question. Does it (preprocessor parameter) override removing stopwords and lowereing case that could be done using stop_words and lowercase parameters?

Response:
You simply define a function that takes a string as input and retuns what is to be preprocessed. So for example a trivial function to uppercase strings would look like this:
```
def preProcess(s):
    return s.upper()
```
Once you have your function made then you just pass it into your TfidfVectorizer object. For example:
```
from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
     'This is the first document.',
     'This is the second second document.',
     'And the third one.',
     'Is this the first document?'
     ]

X = TfidfVectorizer(preprocessor=preProcess)
X.fit(corpus)
X.get_feature_names()
```
Results in:
```
[u'AND', u'DOCUMENT', u'FIRST', u'IS', u'ONE', u'SECOND', u'THE', u'THIRD', u'THIS']
```
This indirectly answers your follow-up question since despite lowercase being set to true, the preprocess function to uppercase overrides it. This is also mentioned in the documentation: preprocessor : callable or None (default) Override the preprocessing (string transformation) stage while preserving the tokenizing and n-grams generation steps.


Question: Do I need to split data when using GridSearchCV? [closed] Closed. This question needs to be more focused. It is not currently accepting answers. Want to improve this question? Guide the asker to update the question so it focuses on a single, specific problem. Narrowing the question will help others answer the question concisely. You may edit the question if you feel you can improve it yourself. If edited, the question will be reviewed and might be reopened. Closed 8 years ago. Improve this question Gridsearhcv uses StratifiedKFold or KFold. So my question is that should I split my data into train and test before using gridsearch, then do fitting only for test data? I am not sure whether it is necessary because cv method already splits the data but I have seen some examples which split data beforehand. Thank you.

Response: GridSearchCV will take the data you give it, split it into Train and CV set and train algorithm searching for the best hyperparameters using the CV set. You can specify different split strategies if you want (for example proportion of split). But when you perform hyperparameter tuning information about dataset still 'leaks' into the algorithm. Hence I would advice to take the following approach: 1) Take your original dataset and hold out some data as a test set (say, 10%) 2) Use grid search on remaining 90%. Split will be done for you by the algorithm here. 3) After you got optimal hyperparameters, test it on the test set from #1 to get final estimate of the performance you can expect on new data.


Question:
ImportError: cannot import name cross_validation I cannot import the cross_validation from sklearn library; I use sklearn version 0.20.0
```
from sklearn import cross_validation
```
later in the code:
```
features_train, features_test, labels_train, labels_test = cross_validation.train_test_split(word_data, authors, test_size=0.1, random_state=42)
```
Error:
```
Traceback (most recent call last):
  File "D:\me\M.Sc\Udacity_ML_course\ud120-projects-  master\naive_bayes\nb_author_id.py", line 16, in <module>
    from email_preprocess import preprocess
  File "../tools/email_preprocess.py", line 8, in <module>
    from sklearn import cross_validation
ImportError: cannot import name cross_validation
```

Response:
This happens because there is no cross_validation object in sklearn. You're likely looking for something more like the cross_validate function. You can access that through
```
from sklearn.model_selection import cross_validate
```
However, you don't need to import any cross-validation software to perform the train-test split, since that will just randomly sample from the data. Try
```
from sklearn.model_selection import train_test_split
```
followed by
```
features_train, features_test, labels_train, labels_test = train_test_split(word_data, authors, test_size=0.1, random_state=42)
```


Question:
What are the arguments for scipy.stats.uniform? I'm trying to create a uniform distribution between two numbers (lower bound and upper bound) in order to feed it to sklearn's ParameterSampler. I am using scipy.stats.uniform in the following format:
```
from scipy.stats import uniform
params = ParameterSampler({'bandwidth':uniform(5,50)}, 20)
```
But when I get the random selections of the 'bandwidth' parameter, they are not all between 5 and 50. Some of them are bigger than 50 by a bit. So my question is what do the arguments in scipy.stats.uniform represent? Are they not a lower bound and upper bound? The documentation shows no arguments so I can't figure it out from that.

Response: The first argument is the lower bound, and the second argument is the range of the distribution. So the example distribution in your question is uniform between 5 and 55. Quoting from the documentation linked in your question: A uniform continuous random variable. This distribution is constant between loc and loc + scale. loc is the first argument and scale is the second argument.


Question:
initial centroids for scikit-learn kmeans clustering if I already have a numpy array that can serve as the initial centroids, how can I properly initialize the kmeans algorithm? I am using the scikit-learn Kmeans class this post (k-means with selected initial centers) indicates that I only need to set n_init=1 if I am using a numpy array as the initial centroids but I am not sure if my initialization is working properly Naftali Harris' excellent visualization page shows what I am trying to do http://www.naftaliharris.com/blog/visualizing-k-means-clustering/ "I'll choose" --> "Packed Circles" --> run kmeans
```
#numpy array of initial centroids
startpts=np.array([[-0.12, 0.939, 0.321, 0.011], [0.0, 0.874, -0.486, 0.862], [0.0, 1.0, 0.0, 0.033], [0.12, 0.939, 0.321, -0.7], [0.0, 1.0, 0.0, -0.203], [0.12, 0.939, -0.321, 0.25], [0.0, 0.874, 0.486, -0.575], [-0.12, 0.939, -0.321, 0.961]], np.float64)

centroids= sk.KMeans(n_clusters=8, init=startpts, n_init=1)

centroids.fit(actual_data_points)

#get the array
centroids_array=centroids.cluster_centers_
```

Response:
Yes, setting initial centroids via init should work. Here's a quote from scikit-learn documentation:
```
init : {‘k-means++’, ‘random’ or an ndarray}

     Method for initialization, defaults to ‘k-means++’:

     If an ndarray is passed, it should be of shape (n_clusters, n_features)
     and gives the initial centers.
```
What is the shape (n_clusters, n_features) referring to? The shape requirement means that init must have exactly n_clusters rows, and the number of elements in each row should match the dimensionality of actual_data_points:
```

```python
init = np.array([[-0.12, 0.939, 0.321, 0.011],
#Output
#                     [0.0, 0.874, -0.486, 0.862],
#                     [0.0, 1.0, 0.0, 0.033],
#                     [0.12, 0.939, 0.321, -0.7],
#                     [0.0, 1.0, 0.0, -0.203],
#                     [0.12, 0.939, -0.321, 0.25],
#                     [0.0, 0.874, 0.486, -0.575],
#                     [-0.12, 0.939, -0.321, 0.961]],
#                    np.float64)
```

```python
init.shape[0] == 8
#Output
#True  # n_clusters
```

```python
init.shape[1] == actual_data_points.shape[1]
#Output
#True  # n_features
#```
#What is n_features? n_features is the dimensionality of your sample. For instance, if you were to cluster points on a 2D plane, n_features would be 2.
```


Question:
I have much more than three elements in every class, but I get this error: "class cannot be less than k=3 in scikit-learn" This is my target (y):
```
target = [7,1,2,2,3,5,4,
      1,3,1,4,4,6,6,
      7,5,7,8,8,8,5,
      3,3,6,2,7,7,1,
      10,3,7,10,4,10,
      2,2,2,7]
```
I do not know why while I'm executing:
```
...
# Split the data set in two equal parts
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.5, random_state=0)

# Set the parameters by cross-validation
tuned_parameters = [{'kernel': ['rbf'], 'gamma': [1e-3, 1e-4],
                 'C': [1, 10, 100, 1000]},
                {'kernel': ['linear'], 'C': [1, 10, 100, 1000]}]

scores = ['precision', 'recall']

for score in scores:
    print("# Tuning hyper-parameters for %s" % score)
    print()

    clf = GridSearchCV(SVC(C=1), tuned_parameters)#scoring non esiste
    # I get an error in the line below
    clf.fit(X_train, y_train, cv=5)
...
```
I get this error:
```
Traceback (most recent call last):
  File "C:\Python27\SVMpredictCROSSeGRID.py", line 232, in <module>
clf.fit(X_train, y_train, cv=5)  #The minimum number of labels for any class cannot be less than k=3.
File "C:\Python27\lib\site-packages\sklearn\grid_search.py", line 354, in fit
return self._fit(X, y)
File "C:\Python27\lib\site-packages\sklearn\grid_search.py", line 372, in _fit
cv = check_cv(cv, X, y, classifier=is_classifier(estimator))
File "C:\Python27\lib\site-packages\sklearn\cross_validation.py", line 1148, in check_cv
cv = StratifiedKFold(y, cv, indices=is_sparse)
File "C:\Python27\lib\site-packages\sklearn\cross_validation.py", line 358, in __init__
" be less than k=%d." % (min_labels, k))
ValueError: The least populated class in y has only 1 members, which is too few. The minimum number of labels for any class cannot be less than k=3.
```

Response: The algorithm requires that there be at least 3 instances for a label in your training set. Although your target array contains at least 3 instances of each label, but when you split the data between training and testing, not all the training labels have 3 instances. You either need to merge some class labels or increase your training samples to solve the problem.


Question: Scikit learn ngram_range purpose in TF-IDF vectorizer? What is the use of ngram_range in vectorizers like countvectorizer and TF-IDF vectorizer. I mean ngram_range(1,1) is for unigram. what it means for ngram_range(1,2) and (2,2)???

Response: ngram_range(1, 2) means unigrams and bigrams, (2, 2) means only bigrams.. Don't you think the docstring is precise enough: The lower and upper boundary of the range of n-values for different n-grams to be extracted. All values of n such that min_n <= n <= max_n will be used. How would you change the docstring to be more helpful?


Question: Getting topic-word distribution from LDA in scikit learn I was wondering if there is a method in the LDA implementation of scikit learn that returns the topic-word distribution. Like the genism show_topics() method. I checked the documentation but didn't find anything.

Response:
Take a look at sklearn.decomposition.LatentDirichletAllocation.components_: components_ : array, [n_topics, n_features] Topic word distribution. components_[i, j] represents word j in topic i. Here's a minimal example:
```
import numpy as np
from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer

data = ['blah blah foo bar', 'foo foo foo foo bar', 'bar bar bar bar foo',
        'foo bar bar bar baz foo', 'foo foo foo bar baz', 'blah banana',
        'cookies candy', 'more text please', 'hey there are more words here',
        'bananas', 'i am a real boy', 'boy', 'girl']

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(data)

vocab = vectorizer.get_feature_names()

n_top_words = 5
k = 2

model = LatentDirichletAllocation(n_topics=k, random_state=100)

id_topic = model.fit_transform(X)

topic_words = {}

for topic, comp in enumerate(model.components_):
    # for the n-dimensional array "arr":
    # argsort() returns a ranked n-dimensional array of arr, call it "ranked_array"
    # which contains the indices that would sort arr in a descending fashion
    # for the ith element in ranked_array, ranked_array[i] represents the index of the
    # element in arr that should be at the ith index in ranked_array
    # ex. arr = [3,7,1,0,3,6]
    # np.argsort(arr) -> [3, 2, 0, 4, 5, 1]
    # word_idx contains the indices in "topic" of the top num_top_words most relevant
    # to a given topic ... it is sorted ascending to begin with and then reversed (desc. now)
    word_idx = np.argsort(comp)[::-1][:n_top_words]

    # store the words most relevant to the topic
    topic_words[topic] = [vocab[i] for i in word_idx]
```
Check out the results:
```
for topic, words in topic_words.items():
    print('Topic: %d' % topic)
    print('  %s' % ', '.join(words))

Topic: 0
  more, blah, here, hey, words
Topic: 1
  foo, bar, blah, baz, boy
```
You should obviously try this code with a much larger body of text, but this is one way to get the most informative words for a given number of topics.


Question:
MemoryError: Unable to allocate MiB for an array with shape and data type, when using anymodel.fit() in sklearn Getting this memory error. But the book/link I am following doesn't get this error. A part of Code:
```
from sklearn.linear_model import SGDClassifier
sgd_clf = SGDClassifier()
sgd_clf.fit(x_train, y_train)
```
Error: MemoryError: Unable to allocate 359. MiB for an array with shape (60000, 784) and data type float64 I also get this error when I try to scale the data using StandardScaler's fit_transfrom But works fine in both if I decrease the size of training set (something like : x_train[:1000] ,y_train[:1000]) Link for the code in the book here. The error I get is in Line 60 and 63 (In [60] and In [63]) The book : Aurélien Géron - Hands-On Machine Learning with Scikit-Learn Keras and Tensorflow 2nd Ed (Page : 149 / 1130) So here's my question : Does this has anything to do with my ram? and what does "Unable to allocate 359" mean? is it the memory size ? Just in case my specs : CPU - ryzen 2400g , ram - 8gb (3.1gb is free when using jupyter notebook)

Response: The message is straight forward, yes, it has to do with the available memory. 359 MiB = 359 * 2^20 bytes = 60000 * 784 * 8 bytes where MiB = Mebibyte = 2^20 bytes, 60000 x 784 are the dimensions of your array and 8 bytes is the size of float64. Maybe the 3.1gb free memory is very fragmented and it is not possible to allocate 359 MiB in one piece? A reboot may be helpful in that case.


Question:
scikit-learn cross validation custom splits for time series data I'd like to use scikit-learn's GridSearchCV to determine some hyper parameters for a random forest model. My data is time dependent and looks something like
```
import pandas as pd

train = pd.DataFrame({'date': pd.DatetimeIndex(['2012-1-1', '2012-9-30', '2013-4-3', '2014-8-16', '2015-3-20', '2015-6-30']),
'feature1': [1.2, 3.3, 2.7, 4.0, 8.2, 6.5],
'feature2': [4, 4, 10, 3, 10, 9],
'target': [1,2,1,3,2,2]})

```python
train
#Output
#        date  feature1  feature2  target
#0 2012-01-01       1.2         4       1
#1 2012-09-30       3.3         4       2
#2 2013-04-03       2.7        10       1
#3 2014-08-16       4.0         3       3
#4 2015-03-20       8.2        10       2
#5 2015-06-30       6.5         9       2
#```
#How can I implement the following cross validation folding technique?
#```
#train:(2012, 2013) - test:(2014)
#train:(2013, 2014) - test:(2015)
#```
#That is, I want to use 2 years of historic observations to train a model and then test its accuracy in the subsequent year.
```

Response:
You just have to pass an iterable with the splits to GridSearchCV. This split should have the following format:
```
[
 (split1_train_idxs, split1_test_idxs),
 (split2_train_idxs, split2_test_idxs),
 (split3_train_idxs, split3_test_idxs),
 ...
]
```
To get the idxs you can do something like this:
```
groups = df.groupby(df.date.dt.year).groups
# {2012: [0, 1], 2013: [2], 2014: [3], 2015: [4, 5]}
sorted_groups = [value for (key, value) in sorted(groups.items())]
# [[0, 1], [2], [3], [4, 5]]

cv = [(sorted_groups[i] + sorted_groups[i+1], sorted_groups[i+2])
      for i in range(len(sorted_groups)-2)]
```
This looks like this:
```
[([0, 1, 2], [3]),  # idxs of first split as (train, test) tuple
 ([2, 3], [4, 5])]  # idxs of second split as (train, test) tuple
```
Then you can do:
```
GridSearchCV(estimator, param_grid, cv=cv, ...)
```


Question:
Suppress scientific notation in sklearn.metrics.plot_confusion_matrix I was trying to plot a confusion matrix nicely, so I followed scikit-learn's newer version 0.22's in built plot confusion matrix function. However, one value of my confusion matrix value is 153, but it appears as 1.5e+02 in the confusion matrix plot: Following the scikit-learn's documentation, I spotted this parameter called values_format, but I do not know how to manipulate this parameter so that it can suppress the scientific notation. My code is as follows.
```
from sklearn import svm, datasets
from sklearn.model_selection import train_test_split
from sklearn.metrics import plot_confusion_matrix

# import some data to play with

X = pd.read_csv("datasets/X.csv")
y = pd.read_csv("datasets/y.csv")

class_names = ['Not Fraud (positive)', 'Fraud (negative)']

# Split the data into a training set and a test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

# Run classifier, using a model that is too regularized (C too low) to see
# the impact on the results
logreg = LogisticRegression()
logreg.fit(X_train, y_train)

np.set_printoptions(precision=2)

# Plot non-normalized confusion matrix
titles_options = [("Confusion matrix, without normalization", None),
                  ("Normalized confusion matrix", 'true')]
for title, normalize in titles_options:
    disp = plot_confusion_matrix(logreg, X_test, y_test,
                                 display_labels=class_names,
                                 cmap=plt.cm.Greens,
                                 normalize=normalize, values_format = '{:.5f}'.format)
    disp.ax_.set_title(title)

    print(title)
    print(disp.confusion_matrix)

plt.show()
```

Response:
Just remove ".format" and the {} brackets from your call parameter declaration:
```
disp = plot_confusion_matrix(logreg, X_test, y_test,
                                 display_labels=class_names,
                                 cmap=plt.cm.Greens,
                                 normalize=normalize, values_format = '.5f')
```
In addition, you can use '.5g' to avoid decimal 0's Taken from source


Question:
Can the Precision, Recall and F1 be the same value? I am working on an ML classification problem and I'm computing the Precision, Recall and F1 using the sklearn library's following import and respective code as shown below.
```
from sklearn.metrics import precision_recall_fscore_support

print(precision_recall_fscore_support(y_test, prob_pos, average='weighted'))
```
Results
```
0.8806451612903226, 0.8806451612903226, 0.8806451612903226
```
Is there a possibility to get the same value for all 3, the precision, recall and F1 for an ML classification problem?

Response: Yes, this is possible. Let's assume binary classification with The trivial solution to Pr = Re = F1 is TP = 0. So we know precision, recall and F1 can have the same value in general. Now, this does not apply to your specific result. If we solve the system of equations, we find another solution: FP = FN. So, if the number of false positives is the same as the number of false negatives, all three metrics have identical values. For multiclass classification problems we have If Pr = Re, again all three metrics are identical.


Question:
sklearn StandardScaler returns all zeros I have a sklearn StandardScaler saved from a previous model and am trying to apply it to new data
```
scaler = myOldStandardScaler
print("ORIG:", X)
print("CLASS:", X.__class__)
X = scaler.fit_transform(X)
print("SCALED:", X)
```
I have three observations each with 2000 features. If I run each observation separately I get an output of all zeros.
```
ORIG: [[  3.19029839e-04   0.00000000e+00   1.90985485e-06 ...,   0.00000000e+00
0.00000000e+00   0.00000000e+00]]
CLASS: <class 'numpy.matrixlib.defmatrix.matrix'>
SCALED: [[ 0.  0.  0. ...,  0.  0.  0.]]
```
But if I append all three observations into one array, I get the results I want
```
ORIG: [[  0.00000000e+00   8.69737728e-08   7.53361877e-06 ...,   0.00000000e+00
0.00000000e+00   0.00000000e+00]
[  9.49627142e-04   0.00000000e+00   0.00000000e+00 ...,   0.00000000e+00
0.00000000e+00   0.00000000e+00]
[  3.19029839e-04   0.00000000e+00   1.90985485e-06 ...,   0.00000000e+00
0.00000000e+00   0.00000000e+00]]
CLASS: <class 'numpy.matrixlib.defmatrix.matrix'>
SCALED: [[-1.07174217  1.41421356  1.37153077 ...,  0.          0.          0.        ]
[ 1.33494964 -0.70710678 -0.98439142 ...,  0.          0.          0.        ]
[-0.26320747 -0.70710678 -0.38713935 ...,  0.          0.          0.        ]]
```
I've seen these two questions: Sklearn's MinMaxScaler only returns zeros Unexpected StandardScaler fit_transform output neither of which have an accepted answer. I've tried: reshaping from (1,n) to (n,1) (this gives incorrect results) converting the array to np.float32 and np.float64 (still all zero) creating an array of an array (again, all zero) creating a np.matrix (again, all zeros) What am I missing? The input to fit_transform is getting the same type, just a different size. How do I get StandardScaler to work with a single observation?

Response:
When you're trying to apply fit_transform method of StandardScaler object to array of size (1, n) you obviously get all zeros, because for each number of array you subtract from it mean of this number, which equal to number and divide to std of this number. If you want to get correct scaling of your array, you should convert it to array with size (n, 1). You can do it this way:
```
import numpy as np

X = np.array([1, -4, 5, 6, -8, 5]) # here should be your X in np.array format
X_transformed = scaler.fit_transform(X[:, np.newaxis])
```
In this case you get Standard scaling for one object by its features, that's not you're looking for. If you want to get scaling by one feature of 3 objects, you should pass to fit_transform method array of size (3, 1) with values of certain feature corresponding to each object.
```
X = np.array([0.00000000e+00, 9.49627142e-04, 3.19029839e-04])
X_transformed = scaler.fit_transform(X[:, np.newaxis]) # you should get
# array([[-1.07174217], [1.33494964], [-0.26320747]]) you're looking for
```
And if you want to work with already fitted StandardScaler object, you shouldn't use fit_transform method, beacuse it refit object with new data. StandardScaler has transform method, which work with single observation:
```
X = np.array([1, -4, 5, 6, -8, 5]) # here should be your X in np.array format
X_transformed = scaler.transform(X.reshape(1, -1))
```


Question: In machine learning, what is definition of “downstream”? I've read the [PCA documentation](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html ) of scikit-learn. [...] improve the predictive accuracy of the downstream estimators [...] What is the definition of “downstream” in machine learning?

Response: The article you are citing refers to whitening and that is a pre-processing step (i.e upstream processing). "Downstream estimators" is a general term referring to the estimators employed after data preprocessing step. Upstream processing (not exclusive list): Binarizer Bucketizer Products Normalizer Tokenizer Scaler Centralizer Downstream estimators (not exclusive list): Logistic regression PCA Neural networks Boosting Gradient descents SVM GNN K-nn K-means Random Forest Hierarchical clustering


Question:
Access standardized residuals, cook's values, hatvalues (leverage) etc. easily in Python? I am looking for influence statistics after fitting a linear regression. In R I can obtain them (e.g.) like this:
```
hatvalues(fitted_model) #hatvalues (leverage)
cooks.distance(fitted_model) #Cook's D values
rstandard(fitted_model) #standardized residuals
rstudent(fitted_model) #studentized residuals
```
etc. How can I obtain the same statistics when using statsmodels in Python after fitting a model like this:
```
#import statsmodels
import statsmodels.api as sm

#Fit linear model to any dataset
model = sm.OLS(Y,X)
results = model.fit()

#Creating a dataframe that includes the studentized residuals
sm.regression.linear_model.OLSResults.outlier_test(results)
```
Edit: See answer below...

Response:
Although the accepted answer is correct, I found it helpful to separately access the statistics as instance attributes of an influence instance (statsmodels.regression.linear_model.OLSResults.get_influence) after I fit my model. This saved me from having to index the summary_frame as I was only interested in one of the statistics and not all of them. So maybe this helps somebody else:
```
import statsmodels.api as sm

#Fit linear model to any dataset
model = sm.OLS(Y,X)
results = model.fit()

#create instance of influence
influence = results.get_influence()

#leverage (hat values)
leverage = influence.hat_matrix_diag

#Cook's D values (and p-values) as tuple of arrays
cooks_d = influence.cooks_distance

#standardized residuals
standardized_residuals = influence.resid_studentized_internal

#studentized residuals
studentized_residuals = influence.resid_studentized_external
```


Question: Sklearn fit vs predict, order of columns matters? Say X1 and X2 are 2 pandas dataframes with the same columns, but possibly in different order. Assume model is some sort of sklearn model, like LassoCV. Say I do model.fit(X1, y), and then model.predict(X2). Is the fact that the columns are in different order a problem, or does model save weights my name of column? Also, same question, but what if X1 and X2 and numpy arrays?

Response:
Yes, I believe it will matter, as sklearn will convert the pandas DataFrame to an array of values (essentially calling X1.values), and not pay attention to the column names. However, it's an easy fix. Just use:
```
X2 = X2[X1.columns]
```
And it will re-order X2's columns to the same order as X1 The same is true of numpy arrays, of course, because it will fit the model on the columns as they are in X1, so when you predict on X2, it will just predict based on the order of the columns in X1 Example: Take these 2 dataframes:
```

```python
X1
#Output
#   a  b
#0  1  5
#1  2  6
#2  3  7
```

```python
X2
#Output
#   b  a
#0  5  3
#1  4  2
#2  6  1
#```
#The model is fit on X1.values:
#```
#array([[1, 5],
#       [2, 6],
#       [3, 7]])
#```
#And you predict on X2.values:
#```
```

```python
X2.values
#Output
#array([[5, 3],
#       [4, 2],
#       [6, 1]])
#```
#There is no way for the model to know that the columns are switched. So switch them manually:
#```
#X2 = X2[X1.columns]
```

```python
X2
#Output
#   a  b
#0  3  5
#1  2  4
#2  1  6
#```
```


Question:
Reusing model fitted by cross_val_score in sklearn using joblib [duplicate] This question already has an answer here: Using sklearn cross_val_score and kfolds to fit and help predict model (1 answer) Closed 3 years ago. The community reviewed whether to reopen this question 3 years ago and left it closed: Original close reason(s) were not resolved I created the following function in python:
```
def cross_validate(algorithms, data, labels, cv=4, n_jobs=-1):
    print "Cross validation using: "
    for alg, predictors in algorithms:
        print alg
        print
        # Compute the accuracy score for all the cross validation folds.
        scores = cross_val_score(alg, data, labels, cv=cv, n_jobs=n_jobs)
        # Take the mean of the scores (because we have one for each fold)
        print scores
        print("Cross validation mean score = " + str(scores.mean()))

        name = re.split('\(', str(alg))
        filename = str('%0.5f' %scores.mean()) + "_" + name[0] + ".pkl"
        # We might use this another time
        joblib.dump(alg, filename, compress=1, cache_size=1e9)
        filenameL.append(filename)
        try:
            move(filename, "pkl")
        except:
            os.remove(filename)

        print
    return
```
I thought that in order to do cross validation, sklearn had to fit your function. However, when I try to use it later (f is the pkl file I saved above in joblib.dump(alg, filename, compress=1, cache_size=1e9)):
```
alg = joblib.load(f)
predictions = alg.predict_proba(train_data[predictors]).astype(float)
```
I get no error in the first line (so it looks like the load is working), but then it tells me NotFittedError: Estimator not fitted, callfitbefore exploiting the model. on the following line. What am I doing wrong? Can't I reuse the model fitted to calculate the cross-validation? I looked at Keep the fitted parameters when using a cross_val_score in scikits learn but either I don't understand the answer, or it is not what I am looking for. What I want is to save the whole model with joblib so that I can the use it later without re-fitting.

Response:
It's not quite correct that cross-validation has to fit your model; rather a k-fold cross validation fits your model k times on partial data sets. If you want the model itself, you actually need to fit the model again on the whole dataset; this actually isn't part of the cross-validation process. So it actually wouldn't be redundant to call
```
alg.fit(data, labels)
```
to fit your model after your cross validation. Another approcach would be rather than using the specialized function cross_val_score, you could think of this as a special case of a cross-validated grid search (with a single point in the parameter space). In this case GridSearchCV will by default refit the model over the entire dataset (it has a parameter refit=True), and also has predict and predict_proba methods in its API.


Question: Setting seed on train_test_split sklearn python is there any way to set seed on train_test_split on python sklearn. I have set the parameter random_state to an integer, but I still can not reproduce the result. Thanks in advance.

Response:
```
from sklearn.model_selection import train_test_split
x = [k for k in range(0, 10)]
y = [k for k in range(0, 10)]
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.4, random_state=11)
print (x_train)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.4, random_state=11)
print (x_train)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.4, random_state=11)
print (x_train)
```
The above code will produce the same result for x_train every time I split the data. It is possible that the randomness is in your dataframe, not train_test_split.


Question:
Python: UnicodeDecodeError: 'utf8' codec can't decode byte I'm reading a bunch of RTF files into python strings. On SOME texts, I get this error:
```
Traceback (most recent call last):
  File "11.08.py", line 47, in <module>
    X = vectorizer.fit_transform(texts)
  File "C:\Python27\lib\site-packages\sklearn\feature_extraction\text.py", line
716, in fit_transform
    X = super(TfidfVectorizer, self).fit_transform(raw_documents)
  File "C:\Python27\lib\site-packages\sklearn\feature_extraction\text.py", line
398, in fit_transform
    term_count_current = Counter(analyze(doc))
  File "C:\Python27\lib\site-packages\sklearn\feature_extraction\text.py", line
313, in <lambda>
    tokenize(preprocess(self.decode(doc))), stop_words)
  File "C:\Python27\lib\site-packages\sklearn\feature_extraction\text.py", line
224, in decode
    doc = doc.decode(self.charset, self.charset_error)
  File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x92 in position 462: invalid
 start byte
```
I've tried: Copying and pasting the text of the files to new files saving the rtf files as txt files Openin the txt files in Notepad++ and choosing 'convert to utf-8' and also setting the encoding to utf-8 Opening the files with Microsoft Word and saving them as new files Nothing works. Any ideas? It's probably not related, but here's the code incase you are wondering:
```
f = open(dir+location, "r")
doc = Rtf15Reader.read(f)
t = PlaintextWriter.write(doc).getvalue()
texts.append(t)
f.close()
vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5, stop_words='english')
X = vectorizer.fit_transform(texts)
```

Response:
This will solve your issues:
```
import codecs

f = codecs.open(dir+location, 'r', encoding='utf-8')
txt = f.read()
```
from that moment txt is in unicode format and you can use it everywhere in your code. If you want to generate UTF-8 files after your processing do:
```
f.write(txt.encode('utf-8'))
```


Question:
Sklearn Chi2 For Feature Selection I'm learning about chi2 for feature selection and came across code like this However, my understanding of chi2 was that higher scores mean that the feature is more independent (and therefore less useful to the model) and so we would be interested in features with the lowest scores. However, using scikit learns SelectKBest, the selector returns the values with the highest chi2 scores. Is my understanding of using the chi2 test incorrect? Or does the chi2 score in sklearn produce something other than a chi2 statistic? See code below for what I mean (mostly copied from above link except for the end)
```
from sklearn.datasets import load_iris
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
import pandas as pd
import numpy as np

# Load iris data
iris = load_iris()

# Create features and target
X = iris.data
y = iris.target

# Convert to categorical data by converting data to integers
X = X.astype(int)

# Select two features with highest chi-squared statistics
chi2_selector = SelectKBest(chi2, k=2)
chi2_selector.fit(X, y)

# Look at scores returned from the selector for each feature
chi2_scores = pd.DataFrame(list(zip(iris.feature_names, chi2_selector.scores_, chi2_selector.pvalues_)), columns=['ftr', 'score', 'pval'])
chi2_scores

# you can see that the kbest returned from SelectKBest
#+ were the two features with the _highest_ score
kbest = np.asarray(iris.feature_names)[chi2_selector.get_support()]
kbest
```

Response: Your understanding is reversed. The null hypothesis for chi2 test is that "two categorical variables are independent". So a higher value of chi2 statistic means "two categorical variables are dependent" and MORE USEFUL for classification. SelectKBest gives you the best two (k=2) features based on higher chi2 values. Thus you need to get those features that it gives, rather that getting the "other features" on the chi2 selector. You are correct to get the chi2 statistic from chi2_selector.scores_ and the best features from chi2_selector.get_support(). It will give you 'petal length (cm)' and 'petal width (cm)' as top 2 features based on chi2 test of independence test. Hope it clarifies this algorithm.


Question:
Implementing ROC Curves for K-NN machine learning algorithm using python and Scikit Learn I am currently trying to implement an ROC Curve for my kNN classification algorithm. I am aware that an ROC Curve is a plot of True Positive Rate vs False Positive Rate, I am just struggling with finding those values from my dataset. I import 'autoimmune.csv' into my python script and run the kNN algorithm on it to output an accuracy value. Scikit-learn.org documentation shows that to generate the TPR and FPR I need to pass in values of y_test and y_scores as shown below:
```
fpr, tpr, threshold = roc_curve(y_test, y_scores)
```
I am just struggling with what I should be using as these values. Thanks for your help in advance and apologies if there is something I have missed as it is my first post here.
```
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.metrics import roc_curve
from sklearn.metrics import auc
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

data = pd.read_csv('./autoimmune.csv')
X = data.drop(columns=['autoimmune'])
y = data['autoimmune'].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

knn = KNeighborsClassifier(n_neighbors = 10)
knn.fit(X_train,y_train)
knn.predict(X_test)[0:10]
knn.score(X_test,y_test)

print("Test set score: {:.4f}".format(knn.score(X_test, y_test)))

knn_cv = KNeighborsClassifier(n_neighbors=10)
cv_scores = cross_val_score(knn_cv, X, y, cv=10)
print(cv_scores)
print('cv_scores mean:{}' .format(np.mean(cv_scores)))

y_scores = cross_val_score(knn_cv, X, y, cv=76)
fpr, tpr, threshold = roc_curve(y_test, y_scores)
roc_auc = auc(fpr, tpr)
print(roc_auc)

plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.title('ROC Curve of kNN')
plt.show()
```

Response:
If you look at the documentation for roc_curve(), you will see the following regarding the y_score parameter: y_score : array, shape = [n_samples] Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers). You can get probability estimates using the predict_proba() method of the KNeighborsClassifier in sklearn. This returns a numpy array with two columns for a binary classification, one each for the negative and positive class. For the roc_curve() function you want to use probability estimates of the positive class, so you can replace your:
```
y_scores = cross_val_score(knn_cv, X, y, cv=76)
fpr, tpr, threshold = roc_curve(y_test, y_scores)
```
with:
```
y_scores = knn.predict_proba(X_test)
fpr, tpr, threshold = roc_curve(y_test, y_scores[:, 1])
```
Notice how you need to take all the rows of the second column with [:, 1] to only select the probability estimates of the positive class. Here's a minimal reproducible example using the Wisconsin breast cancer dataset, since I don't have your autoimmune.csv:
```
from sklearn.datasets import load_breast_cancer
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_curve
from sklearn.metrics import auc
import matplotlib.pyplot as plt

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

knn = KNeighborsClassifier(n_neighbors = 10)
knn.fit(X_train,y_train)

y_scores = knn.predict_proba(X_test)
fpr, tpr, threshold = roc_curve(y_test, y_scores[:, 1])
roc_auc = auc(fpr, tpr)

plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.title('ROC Curve of kNN')
plt.show()
```
This produces the following ROC curve:


Question:
Saving a cross-validation trained model in Scikit I have trained a model in scikit-learn using Cross-Validation and Naive Bayes classifier. How can I persist this model to later run against new instances? Here is simply what I have, I can get the CV scores but I don't know how to have access to the trained model
```
gnb = GaussianNB()
scores = cross_validation.cross_val_score(gnb, data_numpy[0],data_numpy[1], cv=10)
```

Response:
cross_val_score doesn't changes your estimator, and it will not return fitted estimator. It just returns score of estimator of cross validation. To fit your estimator - you should call fit on it explicitly with provided dataset. To save (serialize) it - you can use pickle:
```
# To fit your estimator
gnb.fit(data_numpy[0], data_numpy[1])
# To serialize
import pickle
with open('our_estimator.pkl', 'wb') as fid:
    pickle.dump(gnb, fid)
# To deserialize estimator later
with open('our_estimator.pkl', 'rb') as fid:
    gnb = pickle.load(fid)
```


Question:
How to know what classes are represented in return array from predict_proba in Scikit-learn I'm starting off with Scikit-learn...
```

```python
import sklearn
```

```python
sklearn.__version__
#Output
#'0.13.1'
```

```python
from sklearn import svm
```

```python
model = svm.SVC(probability=True)
```

```python
X = [[1,2,3], [2,3,4]] # feature vectors
```

```python
Y = ['apple', 'orange'] # classes
```

```python
model.fit(X, Y)
```

```python
model.predict_proba([1,2,3])
#Output
#array([[ 0.39097541,  0.60902459]])
#```
#How do I know which class is supposed to be which?
```

Response: The prediction results belong to the classes in this order: model.classes_


Question:
How to estimate the progress of a GridSearchCV from verbose output in Scikit-Learn? Right now I'm running a pretty aggressive grid search. I have n=135 samples and I am running 23 folds using a custom cross-validation train/test list. I have my verbose=2. The following is what I ran:
```
param_test = {"loss":["deviance"],
           'learning_rate':[0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2],
           "min_samples_split": np.linspace(0.1, 0.5, 12),
           "min_samples_leaf": np.linspace(0.1, 0.5, 12),
           "max_depth":[3,5,8],
          "max_features":["log2","sqrt"],
          "min_impurity_split":[5e-6, 1e-7, 5e-7],
          "criterion": ["friedman_mse",  "mae"],
           "subsample":[0.5, 0.618, 0.8, 0.85, 0.9, 0.95, 1.0],
          "n_estimators":[10]}

Mod_gsearch = GridSearchCV(estimator = GradientBoostingClassifier(),
                           param_grid = param_test, scoring="accuracy",n_jobs=32, iid=False, cv=cv_indices, verbose=2)
```
I took a look at the verbose output in stdout:
```
$head gridsearch.o8475533
Fitting 23 folds for each of 254016 candidates, totalling 5842368 fits
```
Based on this, it looks like there are 5842368 permutations of cross-validation pairs using my grid params.
```
$ grep -c  "[CV]" gridsearch.o8475533
7047332
```
It looks like there are around 7 million cross-validations that have been done so far but that's more than the 5842368 total fits...
```
7047332/5842368 = 1.2062458236
```
Then when I look at the stderr file:
```
$ cat ./gridsearch.e8475533
[Parallel(n_jobs=32)]: Done 132 tasks      | elapsed:    1.2s
[Parallel(n_jobs=32)]: Done 538 tasks      | elapsed:    2.8s
[Parallel(n_jobs=32)]: Done 1104 tasks      | elapsed:    4.8s
[Parallel(n_jobs=32)]: Done 1834 tasks      | elapsed:    7.9s
[Parallel(n_jobs=32)]: Done 2724 tasks      | elapsed:   11.6s
...
[Parallel(n_jobs=32)]: Done 3396203 tasks      | elapsed: 250.2min
[Parallel(n_jobs=32)]: Done 3420769 tasks      | elapsed: 276.5min
[Parallel(n_jobs=32)]: Done 3447309 tasks      | elapsed: 279.3min
[Parallel(n_jobs=32)]: Done 3484240 tasks      | elapsed: 282.3min
[Parallel(n_jobs=32)]: Done 3523550 tasks      | elapsed: 285.3min
```
My goal: How can I know the progress of my gridsearch with respect to the total time it may take? What I'm confused about: What is the relationship between [CV] lines in stdout, total # of fits in stdout, and tasks in stderr?

Response: Math is simple, but a little misleading at a first sight: When each task is started logging mechanism yields a '[CV] ...' line to stdout noting about starting of execution and after task ends - another line with the addition of spent time for a particular task (in the end of the line). Additionally, with some time intervals, logging mechanism writes a progress bar to stderr (or if you set verbose to >50 to stdout) indicating a number of completed task out of total tasks (fits) and total currently spent time, like that one: [Parallel(n_jobs=32)]: Done 2724 tasks | elapsed: 11.6s For your case, you have 5842368 total fits, i.e. tasks. You counted 7047332 of '[CV] ...' which is around 7047332/2 = 3523666 finished tasks and progress bar shows exactly how many tasks are completed - 3523550 (around - because some tasks could start, but not end at the time of counting).


Question:
Accuracy score in pyTorch LSTM I have been running this LSTM tutorial on the wikigold.conll NER data set training_data contains a list of tuples of sequences and tags, for example:
```
training_data = [
    ("They also have a song called \" wake up \"".split(), ["O", "O", "O", "O", "O", "O", "I-MISC", "I-MISC", "I-MISC", "I-MISC"]),
    ("Major General John C. Scheidt Jr.".split(), ["O", "O", "I-PER", "I-PER", "I-PER"])
]
```
And I wrote down this function
```
def predict(indices):
    """Gets a list of indices of training_data, and returns a list of predicted lists of tags"""
    for index in indicies:
        inputs = prepare_sequence(training_data[index][0], word_to_ix)
        tag_scores = model(inputs)
        values, target = torch.max(tag_scores, 1)
        yield target
```
This way I can get the predicted labels for specific indices in the training data. However, how do I evaluate the accuracy score across all training data. Accuracy being, the amount of words correctly classified across all sentences divided by the word count. This is what I came up with, which is extremely slow and ugly:
```
y_pred = list(predict([s for s, t in training_data]))
y_true = [t for s, t in training_data]
c=0
s=0
for i in range(len(training_data)):
    n = len(y_true[i])
    #super ugly and ineffiicient
    s+=(sum(sum(list(y_true[i].view(-1, n) == y_pred[i].view(-1, n).data))))
    c+=n

print ('Training accuracy:{a}'.format(a=float(s)/c))
```
How can this be done efficiently in pytorch ? P.S: I've been trying to use sklearn's accuracy_score unsuccessfully

Response:
I would use numpy in order to not iterate the list in pure python. The results are the same, but it runs much faster
```
def accuracy_score(y_true, y_pred):
    y_pred = np.concatenate(tuple(y_pred))
    y_true = np.concatenate(tuple([[t for t in y] for y in y_true])).reshape(y_pred.shape)
    return (y_true == y_pred).sum() / float(len(y_true))
```
And this is how to use it:
```
#original code:
y_pred = list(predict([s for s, t in training_data]))
y_true = [t for s, t in training_data]
#numpy accuracy score
print(accuracy_score(y_true, y_pred))
```


Question:
Multi-class multi-label confusion matrix with Sklearn I am working with a multi-class multi-label output from my classifier. The total number of classes is 14 and instances can have multiple classes associated. For example:
```
y_true = np.array([[0,0,1], [1,1,0],[0,1,0])
y_pred = np.array([[0,0,1], [1,0,1],[1,0,0])
```
The way I am making my confusion matrix right now:
```
matrix = confusion_matrix(y_true.argmax(axis=1), y_pred.argmax(axis=1))
print(matrix)
```
Which gives an output like:
```
[[ 79   0   0   0  66   0   0 151   1   8   0   0   0   0]
 [  4   0   0   0  11   0   0  27   0   0   0   0   0   0]
 [ 14   0   0   0  21   0   0  47   0   1   0   0   0   0]
 [  1   0   0   0   4   0   0  25   0   0   0   0   0   0]
 [ 18   0   0   0  50   0   0  63   0   3   0   0   0   0]
 [  4   0   0   0   3   0   0  19   0   0   0   0   0   0]
 [  2   0   0   0   3   0   0  11   0   2   0   0   0   0]
 [ 22   0   0   0  20   0   0 138   1   5   0   0   0   0]
 [ 12   0   0   0   9   0   0  38   0   1   0   0   0   0]
 [ 10   0   0   0   3   0   0  40   0   4   0   0   0   0]
 [  3   0   0   0   3   0   0  14   0   3   0   0   0   0]
 [  0   0   0   0   2   0   0   3   0   0   0   0   0   0]
 [  2   0   0   0  11   0   0  32   0   0   0   0   0   0]
 [  0   0   0   0   0   0   0   3   0   0   0   0   0   7]]
```
Now, I am not sure if the confusion matrix from sklearn is capable of handling multi-label multi-class data. Could someone help me with this?

Response:
Now you can use (version 0.21) sklearn.metrics.multilabel_confusion_matrix https://scikit-learn.org/stable/modules/generated/sklearn.metrics.multilabel_confusion_matrix.html We try to predict two labels for each example
```
import sklearn.metrics as skm
y_true = np.array([
                [0,0], [0,1], [1,1], [0,1], [0,1], [1,1]
              ])
 y_pred = np.array([
                [1,1], [0,1], [0,1], [1,0], [0,1], [1,1]
              ])

 cm = skm.multilabel_confusion_matrix(y_true, y_pred)
 print(cm)
 print( skm.classification_report(y_true,y_pred))
```
Confusion matrix for labels:
```
[[[2 2]
  [1 1]]

 [[0 1]
  [1 4]]]
```
Classification report:
```
precision    recall  f1-score   support

         0       0.33      0.50      0.40         2
         1       0.80      0.80      0.80         5

micro avg        0.62      0.71      0.67         7
macro avg        0.57      0.65      0.60         7
weighted avg     0.67      0.71      0.69         7
samples avg      0.67      0.58      0.61         7
```


Question:
AttributeError:'LinearSVC' object has no attribute 'predict_proba' I am trying to use LinearSVC classifier Update: Added imports
```
import nltk
from nltk.tokenize import word_tokenize
from nltk.classify.scikitlearn import SklearnClassifier
from sklearn.svm import LinearSVC, SVC

LinearSVC_classifier = SklearnClassifier(LinearSVC())
LinearSVC_classifier.train(featuresets)
```
But when I am trying to classify it with probabilities
```
LinearSVC_classifier.prob_classify(feats)
```
AttributeError occurs:
```
AttributeError:'LinearSVC' object has no attribute 'predict_proba'
```
I checked sklearn documentation, it tells that this function exist. How to fix that?

Response:
According to sklearn documentation , the method 'predict_proba' is not defined for 'LinearSVC' Workaround:
```
LinearSVC_classifier = SklearnClassifier(SVC(kernel='linear',probability=True))
```
Use SVC with linear kernel, with probability argument set to True. Just as explained in here .


Question:
what is Gridsearch.cv_results_ , could any explain all the things in that i.e mean_test_score etc .? I am doing hyperparameter tuning with GridSearchCV for Decision Trees. I have fit the model and I am trying to find what does exactly Gridsearch.cv_results_ gives. I have read the documentation but still its not clear. Could anyone explain this attribute? my code is below :
```
depth={"max_depth":[1,5,10,50,100,500,1000],
       "min_samples_split":[5,10,100,500]}

DTC=DecisionTreeClassifier(class_weight="balanced")

DTC_Grid=GridSearchCV(DTC,param_grid=depth , cv=3, scoring='roc_auc')
DTC_Bow=DTC_Grid.fit(xtrain_bow,ytrain_bow)
```

Response:
DTC_Bow.cv_results_ returns a dictionary of all the evaluation metrics from the gridsearch. To visualize it properly, you can do
```
pd.DataFrame(DTC_Bow.cv_results_)
```
In your case, this should return a dataframe with 28 rows (7 choices for max_depth times 4 choices for min_samples_split). Each row of this dataframe gives the gridsearch metrics for one combination of these two parameters. Remember the goal of a gridsearch is to select which combination of parameters will have the best performance metrics. This is the purpose of cv_results_. You should have one column called param_max_depth and another called param_min_samples_leaf referencing the value of the parameter for each row. The combination of two is summarized as a dictionary in the column params. Now to the metrics. Default value for return_train_score was True up until now but they will change it to False in version 0.21. If you want the train metrics, set it to True. But usually, what you are interested in are the test metrics. The main column is mean_test_score. This is the average of columns split_0_test_score, split_1_test_score, split_2_test_score (because you are doing a 3 fold split in your gridsearch). If you do DTC_Bow.best_score_ this will return the max value of the column mean_test_score. Column rank_test_score ranks all parameter combinations by the values of mean_test_score. You might also want to look at std_test_score which is the standard deviation of split_0_test_score, split_1_test_score, split_2_test_score. This might be of interest if you want to see how consistently your set of parameters is performing on your hold-out data. As mentioned, you can have the metrics on the train set as well provided you set return_train_score = True. Finally, there are also time columns, that tell you how much time it took for each row. It measures how much time it took to train the model (mean_fit_time, std_fit_time) and to evaluate it (mean_score_time, std_score_time). This is just a FYI, usually, unless time is a bottleneck, you would not really look at these metrics.


Question:
deprecation error in sklearn about empty array without any empty array in my code I am just playing around encoding and decoding but I get this error from sklearn: Warning (from warnings module): File "C:\Python36\lib\site-packages\sklearn\preprocessing\label.py", line 151 if diff: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use array.size > 0 to check that an array is not empty. Here is the full code, you can run it yourself in python 3+ My question is why is it saying I use an empty array as I clearly don't in my code, thanks for taking your time to answer my question.
```
### label encoding ###

import numpy as np
from sklearn import preprocessing

# Sample input labels
input_labels = ["red", "black", "red", "green",\
                "black", "yellow", "white"]

# Create label encoder abd fit the label
encoder = preprocessing.LabelEncoder()
encoder.fit(input_labels)

# Print the mapping
print("\nLabel mapping:")
for i, item in enumerate(encoder.classes_):
    print(item, "-->", i)

# Encode a set of labels using encoder
test_labels = ["green", "red", "black"]
encoded_values = encoder.transform(test_labels)
print("\nLabels =", test_labels)
print("Encoded values =", list(encoded_values))

# Decode a set of values using the encoder
encoded_values = [3, 0, 4, 1]
decoded_list = encoder.inverse_transform(encoded_values)
print("\nEncoded values =", encoded_values)
print("Decoded labels=", list(decoded_list))
```

Response:
TLDR: You can ignore the warning. It is caused by sklearn doing something internally that is not quite ideal. The warning is actually caused by numpy, which deprecated truth testing on empty arrays: The long and short is that truth-testing on empty arrays is dangerous, misleading, and not in any way useful, and should be deprecated. This means one is not supposed to do something like if array: to check if array is empty. However, sklearn does this in the 0.19.1 release:
```
diff = np.setdiff1d(y, np.arange(len(self.classes_)))
    if diff:
        raise ValueError("y contains new labels: %s" % str(diff))
```
Because your version of numpy is recent enough it complains and issues a warning. The problem has been fixed in sklearn's current master branch, so I'd expect the fix to be included in the next release.


Question:
CountVectorizer: "I" not showing up in vectorized text I'm new to scikit-learn, and currently studying Naïve Bayes (Multinomial). Right now, I'm working on vectorizing text from sklearn.feature_extraction.text, and for some reason, when I vectorize some text, the word "I" doesn't show up in the outputted array. Code:
```
x_train = ['I am a Nigerian hacker', 'I like puppies']

# convert x_train to vectorized text
vectorizer_train = CountVectorizer(min_df=0)
vectorizer_train.fit(x_train)
x_train_array = vectorizer_train.transform(x_train).toarray()

# print vectorized text, feature names
print x_train_array
print vectorizer_train.get_feature_names()
```
Output:
```
1 1 0 1 0
0 0 1 0 1
[u'am', u'hacker', u'like', u'nigerian', u'puppies']
```
Why doesn't "I" seem to show up in the feature names? When I change it to "Ia" or something else like that, it does show up.

Response:
This is caused by the default token_pattern for CountVectorizer, which removes tokens of a single character:
```

```python
vectorizer_train
#Output
#CountVectorizer(analyzer=u'word', binary=False, charset=None,
#        charset_error=None, decode_error=u'strict',
#        dtype=<type 'numpy.int64'>, encoding=u'utf-8', input=u'content',
#        lowercase=True, max_df=1.0, max_features=None, min_df=0,
#        ngram_range=(1, 1), preprocessor=None, stop_words=None,
#        strip_accents=None, token_pattern=u'(?u)\\b\\w\\w+\\b',
#        tokenizer=None, vocabulary=None)
```

```python
pattern = re.compile(vectorizer_train.token_pattern, re.UNICODE)
```

```python
print(pattern.match("I"))
#Output
#None
#```
#To retain "I", use a different pattern, e.g.
#```
```

```python
vectorizer_train = CountVectorizer(min_df=0, token_pattern=r"\b\w+\b")
```

```python
vectorizer_train.fit(x_train)
#Output
#CountVectorizer(analyzer=u'word', binary=False, charset=None,
#        charset_error=None, decode_error=u'strict',
#        dtype=<type 'numpy.int64'>, encoding=u'utf-8', input=u'content',
#        lowercase=True, max_df=1.0, max_features=None, min_df=0,
#        ngram_range=(1, 1), preprocessor=None, stop_words=None,
#        strip_accents=None, token_pattern='\\b\\w+\\b', tokenizer=None,
#        vocabulary=None)
```

```python
vectorizer_train.get_feature_names()
#Output
#[u'a', u'am', u'hacker', u'i', u'like', u'nigerian', u'puppies']
#```
#Note that the non-informative word "a" is now also retained.
```


Question:
Should a pandas dataframe column be converted in some way before passing it to a scikit learn regressor? I have a pandas dataframe and passing df[list_of_columns] as X and df[[single_column]] as Y to a Random Forest regressor. What does the following warnning mean and what should be done to resolve it?
```
DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().   probas = cfr.fit(trainset_X, trainset_Y).predict(testset_X)
```

Response: Simply check the shape of your Y variable, it should be a one-dimensional object, and you are probably passing something with more (possibly trivial) dimensions. Reshape it to the form of list/1d array.


Question:
AxisError: axis 1 is out of bounds for array of dimension 1 when calculating AUC I have a classification problem where I have the pixels values of an 8x8 image and the number the image represents and my task is to predict the number('Number' attribute) based on the pixel values using RandomForestClassifier. The values of the number values can be 0-9.
```
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score

forest_model = RandomForestClassifier(n_estimators=100, random_state=42)
forest_model.fit(train_df[input_var], train_df[target])
test_df['forest_pred'] = forest_model.predict_proba(test_df[input_var])[:,1]
roc_auc_score(test_df['Number'], test_df['forest_pred'], average = 'macro', multi_class="ovr")
```
Here it throws an AxisError.
```
Traceback (most recent call last):
  File "dap_hazi_4.py", line 44, in
    roc_auc_score(test_df['Number'], test_df['forest_pred'], average = 'macro', multi_class="ovo")
  File "/home/balint/.local/lib/python3.6/site-packages/sklearn/metrics/_ranking.py", line 383, in roc_auc_score
    multi_class, average, sample_weight)
  File "/home/balint/.local/lib/python3.6/site-packages/sklearn/metrics/_ranking.py", line 440, in _multiclass_roc_auc_score
    if not np.allclose(1, y_score.sum(axis=1)):
  File "/home/balint/.local/lib/python3.6/site-packages/numpy/core/_methods.py", line 38, in _sum
    return umr_sum(a, axis, dtype, out, keepdims, initial, where)

AxisError: axis 1 is out of bounds for array of dimension 1
```

Response:
The error is due to multi-class problem that you are solving as others suggested. All you need to do is instead of predicting the class, you need to predict the probabilities. I had this same problem before, doing this solves it. Here is how to do it -
```
# you might be predicting the class this way
pred = clf.predict(X_valid)

# change it to predict the probabilities which solves the AxisError problem.
pred_prob = clf.predict_proba(X_valid)
roc_auc_score(y_valid, pred_prob, multi_class='ovr')
0.8164900342274142

# shape before
pred.shape
(256,)
pred[:5]
array([1, 2, 1, 1, 2])

# shape after
pred_prob.shape
(256, 3)
pred_prob[:5]
array([[0.  , 1.  , 0.  ],
       [0.02, 0.12, 0.86],
       [0.  , 0.97, 0.03],
       [0.  , 0.8 , 0.2 ],
       [0.  , 0.42, 0.58]])
```


Question: How to calculate the 99% confidence interval for the slope in a linear regression model in python? We have following linear regression: y ~ b0 + b1 * x1 + b2 * x2. I know that regress function in Matlab does calculate it, but numpy's linalg.lstsq doesn't (https://docs.scipy.org/doc/numpy-dev/user/numpy-for-matlab-users.html).

Response:
StatsModels' RegressionResults has a conf_int() method. Here an example using it (minimally modified version of their Ordinary Least Squares example):
```
import numpy as np, statsmodels.api as sm

nsample = 100
x = np.linspace(0, 10, nsample)
X = np.column_stack((x, x**2))
beta = np.array([1, 0.1, 10])
e = np.random.normal(size=nsample)

X = sm.add_constant(X)
y = np.dot(X, beta) + e

mod = sm.OLS(y, X)
res = mod.fit()
print res.conf_int(0.01)   # 99% confidence interval
```


Question:
Can sklearn DecisionTreeClassifier truly work with categorical data? While working with the DecisionTreeClassifier I visualized it using graphviz, and I have to say, to my astonishment, it seems it takes categorical data and uses it as continuous data. All my features are categorical and for example you can see the following tree (please note that the first feature, X[0], has 6 possible values 0, 1, 2, 3, 4, 5: From what I found here the class uses a tree class which is a binary tree, so it is a limitation in sklearn. Anyone knows a way that I am missing to use the tree categorically? (I know it is not better for the task but as I need categories currently I am using one hot vectors on the data). A sample of the original data looks like this:
```
f1 f2 f3  f4  f5  f6  f7  f8  f9  f10  c1  c2  c3
0  C  S  O   1   2   1   1   2   1    2   0   0   0
1  D  S  O   1   3   1   1   2   1    2   0   0   0
2  C  S  O   1   3   1   1   2   1    1   0   0   0
3  D  S  O   1   3   1   1   2   1    2   0   0   0
4  D  A  O   1   3   1   1   2   1    2   0   0   0
5  D  A  O   1   2   1   1   2   1    2   0   0   0
6  D  A  O   1   2   1   1   2   1    1   0   0   0
7  D  A  O   1   2   1   1   2   1    2   0   0   0
8  D  K  O   1   3   1   1   2   1    2   0   0   0
9  C  R  O   1   3   1   1   2   1    1   0   0   0
```
where X[0] = f1 and I encoded strings to integers as sklearn does not accept strings.

Response: Well, I am surprised, but it turns out that sklearn's decision tree cannot handle categorical data indeed. There is a Github issue on this (#4899) from June 2015, but it is still open (UPDATE: it is now closed, but continued in #12866, so the issue is still not resolved). The problem with coding categorical variables as integers, as you have done here, is that it imposes an order on them, which may or may not be meaningful, depending on the case; for example, you could encode ['low', 'medium', 'high'] as [0, 1, 2], since 'low' < 'medium' < 'high' (we call these categorical variables ordinal), although you are still implicitly making the additional (and possibly undesired) assumption that the distance between 'low' and 'medium' is the same with the distance between 'medium' and 'high' (of no impact in decision trees, but of importance e.g. in k-nn and clustering). But this approach fails completely in cases like, say, ['red','green','blue'] or ['male','female'], since we cannot claim any meaningful relative order between them. So, for non-ordinal categorical variables, the way to properly encode them for use in sklearn's decision tree is to use the OneHotEncoder module. The Encoding categorical features section of the user's guide might also be helpful.


Question:
How to get Predictions with XGBoost and XGBoost using Scikit-Learn Wrapper to match? I am new to XGBoost in Python so I apologize if the answer here is obvious, but I am trying to take a panda dataframe and get XGBoost in Python to give me the same predictions I get when I use the Scikit-Learn wrapper for the same exercise. So far I've been unable to do so. Just to give an example, here I take the boston dataset, convert to a panda dataframe, train on the first 500 observations of the dataset and then predict the last 6. I do it with XGBoost first and then with the Scikit-Learn wrapper and I get different predictions even though I've set the parameters of the model to be the same. Specifically the array predictions looks very different from the array predictions2 (see code below). Any help would be much appreciated!
```
from sklearn import datasets
import pandas as pd
import xgboost as xgb
from xgboost.sklearn import XGBClassifier
from xgboost.sklearn import XGBRegressor

### Use the boston data as an example, train on first 500, predict last 6
boston_data = datasets.load_boston()
df_boston = pd.DataFrame(boston_data.data,columns=boston_data.feature_names)
df_boston['target'] = pd.Series(boston_data.target)

#### Code using XGBoost
Sub_train = df_boston.head(500)
target = Sub_train["target"]
Sub_train = Sub_train.drop('target', axis=1)

Sub_predict = df_boston.tail(6)
Sub_predict = Sub_predict.drop('target', axis=1)

xgtrain = xgb.DMatrix(Sub_train.as_matrix(), label=target.tolist())
xgtest = xgb.DMatrix(Sub_predict.as_matrix())

params = {'booster': 'gblinear', 'objective': 'reg:linear',
      'max_depth': 2, 'learning_rate': .1, 'n_estimators': 500,    'min_child_weight': 3, 'colsample_bytree': .7,
      'subsample': .8, 'gamma': 0, 'reg_alpha': 1}

model = xgb.train(dtrain=xgtrain, params=params)

predictions = model.predict(xgtest)

#### Code using Sk learn Wrapper for XGBoost
model = XGBRegressor(learning_rate =.1, n_estimators=500,
max_depth=2, min_child_weight=3, gamma=0,
subsample=.8, colsample_bytree=.7, reg_alpha=1,
objective= 'reg:linear')

target = "target"

Sub_train = df_boston.head(500)
Sub_predict = df_boston.tail(6)
Sub_predict = Sub_predict.drop('target', axis=1)

Ex_List = ['target']

predictors = [i for i in Sub_train.columns if i not in Ex_List]

model = model.fit(Sub_train[predictors],Sub_train[target])

predictions2 = model.predict(Sub_predict)
```

Response:
Please look at this answer here xgboost.train will ignore parameter n_estimators, while xgboost.XGBRegressor accepts. In xgboost.train, boosting iterations (i.e. n_estimators) is controlled by num_boost_round(default: 10) It suggests to remove n_estimators from params supplied to xgb.train and replace it with num_boost_round. So change your params like this:
```
params = {'objective': 'reg:linear',
      'max_depth': 2, 'learning_rate': .1,
      'min_child_weight': 3, 'colsample_bytree': .7,
      'subsample': .8, 'gamma': 0, 'alpha': 1}
```
And train xgb.train like this:
```
model = xgb.train(dtrain=xgtrain, params=params,num_boost_round=500)
```
And you will get same results. Alternatively, keep the xgb.train as it is and change the XGBRegressor like this:
```
model = XGBRegressor(learning_rate =.1, n_estimators=10,
                     max_depth=2, min_child_weight=3, gamma=0,
                     subsample=.8, colsample_bytree=.7, reg_alpha=1,
                     objective= 'reg:linear')
```
Then also you will get same results.


Question:
Scikit learn - fit_transform on the test set I am struggling to use Random Forest in Python with Scikit learn. My problem is that I use it for text classification (in 3 classes - positive/negative/neutral) and the features that I extract are mainly words/unigrams, so I need to convert these to numerical features. I found a way to do it with DictVectorizer's fit_transform:
```
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import classification_report
from sklearn.feature_extraction import DictVectorizer

vec = DictVectorizer(sparse=False)
rf = RandomForestClassifier(n_estimators = 100)
trainFeatures1 = vec.fit_transform(trainFeatures)

# Fit the training data to the training output and create the decision trees
rf = rf.fit(trainFeatures1.toarray(), LabelEncoder().fit_transform(trainLabels))

testFeatures1 = vec.fit_transform(testFeatures)
# Take the same decision trees and run on the test data
Output = rf.score(testFeatures1.toarray(), LabelEncoder().fit_transform(testLabels))

print "accuracy: " + str(Output)
```
My problem is that the fit_transform method is working on the train dataset, which contains around 8000 instances, but when I try to convert my test set to numerical features too, which is around 80000 instances, I get a memory error saying that:
```
testFeatures1 = vec.fit_transform(testFeatures)
File "C:\Python27\lib\site-packages\sklearn\feature_extraction\dict_vectorizer.py", line 143, in fit_transform
return self.transform(X)
File "C:\Python27\lib\site-packages\sklearn\feature_extraction\dict_vectorizer.py", line 251, in transform
Xa = np.zeros((len(X), len(vocab)), dtype=dtype)
MemoryError
```
What could possibly cause this and is there any workaround? Many thanks!

Response:
You are not supposed to do fit_transform on your test data, but only transform. Otherwise, you will get different vectorization than the one used during training. For the memory issue, I recommend TfIdfVectorizer, which has numerous options of reducing the dimensionality (by removing rare unigrams etc.). UPDATE If the only problem is fitting test data, simply split it to small chunks. Instead of something like
```
x=vect.transform(test)
eval(x)
```
you can do
```
K=10
for i in range(K):
    size=len(test)/K
    x=vect.transform(test[ i*size : (i+1)*size ])
    eval(x)
```
and record results/stats and analyze them afterwards. in particular
```
predictions = []

K=10
for i in range(K):
    size=len(test)/K
    x=vect.transform(test[ i*size : (i+1)*size ])
    predictions += rf.predict(x) # assuming it retuns a list of labels, otherwise - convert it to list

print accuracy_score( predictions, true_labels )
```


Question: What to do first: Feature Selection or Model Parameters Setting? This is more of a theoretical question. I'm working with the scikit-learn package to perform some NLP task. Sklearn provides many methods to perform both feature selection and setting of a model parameters. I'm wondering what I should do first. If I use univariate feature selection, it's pretty obvious that I should do feature selection first and, with the selected features, I then tunne the parameters of the estimator. But what if I want to use recursive feature elimination? Should I first set the parameters with grid search using ALL the original features and just then perform feature selection? Or perhaps I should select the features first (with the estimator's default parameters) and then set the parameters with the selected features? EDIT I'm having pretty much the same problem stated here. By that time, there wasn't a solution to it. Does anyone know if it exists one now?

Response: Personally I think RFE is overkill and too expensive in most cases. If you want to do feature selection on linear models, use univariate feature selection, for instance with chi2 tests or L1 or L1 + L2 regularized models with grid searched regularization parameter (usually named C or alpha in sklearn models). For highly non-linear problems with a lot of samples you should try RandomForestClassifier, ExtraTreesClassifier or GBRT models and grid searched parameters selection (possibly using OOB score estimates) and use the compute_importances switch to find a ranking of features by importance and use that for feature selection. For highly non-linear problems with few samples I don't think there is a solution. You must be doing neurosciences :)


Question:
Difference between using train_test_split and cross_val_score in sklearn.cross_validation I have a matrix with 20 columns. The last column are 0/1 labels. The link to the data is here. I am trying to run random forest on the dataset, using cross validation. I use two methods of doing this: using sklearn.cross_validation.cross_val_score using sklearn.cross_validation.train_test_split I am getting different results when I do what I think is pretty much the same exact thing. To exemplify, I run a two-fold cross validation using the two methods above, as in the code below.
```
import csv
import numpy as np
import pandas as pd
from sklearn import ensemble
from sklearn.metrics import roc_auc_score
from sklearn.cross_validation import train_test_split
from sklearn.cross_validation import cross_val_score

#read in the data
data = pd.read_csv('data_so.csv', header=None)
X = data.iloc[:,0:18]
y = data.iloc[:,19]

depth = 5
maxFeat = 3

result = cross_val_score(ensemble.RandomForestClassifier(n_estimators=1000, max_depth=depth, max_features=maxFeat, oob_score=False), X, y, scoring='roc_auc', cv=2)

result
# result is now something like array([ 0.66773295,  0.58824739])

xtrain, xtest, ytrain, ytest = train_test_split(X, y, test_size=0.50)

RFModel = ensemble.RandomForestClassifier(n_estimators=1000, max_depth=depth, max_features=maxFeat, oob_score=False)
RFModel.fit(xtrain,ytrain)
prediction = RFModel.predict_proba(xtest)
auc = roc_auc_score(ytest, prediction[:,1:2])
print auc    #something like 0.83

RFModel.fit(xtest,ytest)
prediction = RFModel.predict_proba(xtrain)
auc = roc_auc_score(ytrain, prediction[:,1:2])
print auc    #also something like 0.83
```
My question is: why am I getting different results, ie, why is the AUC (the metric I am using) higher when I use train_test_split? Note: When I using more folds (say 10 folds), there appears to be some kind of pattern in my results, with the first calculation always giving me the highest AUC. In the case of the two-fold cross validation in the example above, the first AUC is always higher than the second one; it's always something like 0.70 and 0.58. Thanks for your help!

Response: When using cross_val_score, you'll frequently want to use a KFolds or StratifiedKFolds iterator: http://scikit-learn.org/0.10/modules/cross_validation.html#computing-cross-validated-metrics http://scikit-learn.org/0.10/modules/generated/sklearn.cross_validation.KFold.html#sklearn.cross_validation.KFold By default, cross_val_score will not randomize your data, which can produce odd results like this if you're data isn't random to begin with. The KFolds iterator has a random state parameter: http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.KFold.html So does train_test_split, which does randomize by default: http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.train_test_split.html Patterns like what you described are usually a result of a lack of randomnesss in the train/test set.


Question: Scikit and Pandas: Fitting Large Data How do I use scikit-learn to train a model on a large csv data (~75MB) without running into memory problems? I'm using IPython notebook as the programming environment, and pandas+sklearn packages to analyze data from kaggle's digit recognizer tutorial. The data is available on the webpage , link to my code , and here is the error message: KNeighborsClassifier is used for the prediction. Problem: "MemoryError" occurs when loading large dataset using read_csv function. To bypass this problem temporarily, I have to restart the kernel, which then read_csv function successfully loads the file, but the same error occurs when I run the same cell again. When the read_csv function loads the file successfully, after making changes to the dataframe, I can pass the features and labels to the KNeighborsClassifier's fit() function. At this point, similar memory error occurs. I tried the following: Iterate through the CSV file in chunks, and fit the data accordingly, but the problem is that the predictive model is overwritten every time for a chunk of data. What do you think I can do to successfully train my model without running into memory problems?

Response: Note: when you load the data with pandas it will create a DataFrame object where each column has an homogeneous datatype for all the rows but 2 columns can have distinct datatypes (e.g. integer, dates, strings). When you pass a DataFrame instance to a scikit-learn model it will first allocate a homogeneous 2D numpy array with dtype np.float32 or np.float64 (depending on the implementation of the models). At this point you will have 2 copies of your dataset in memory. To avoid this you could write / reuse a CSV parser that directly allocates the data in the internal format / dtype expected by the scikit-learn model. You can try numpy.loadtxt for instance (have a look at the docstring for the parameters). Also if you data is very sparse (many zero values) it will be better to use a scipy.sparse datastructure and a scikit-learn model that can deal with such an input format (check the docstrings to know). However the CSV format itself is not very well suited for sparse data and I am not sure there exist a direct CSV-to-scipy.sparse parser. Edit: for reference KNearestNeighborsClassifer allocate temporary distances array with shape (n_samples_predict, n_samples_train) which is very wasteful when only (n_samples_predict, n_neighbors) is needed instead. This issue can be tracked here: https://github.com/scikit-learn/scikit-learn/issues/325


Question: How to obtain information gain from a scikit-learn DecisionTreeClassifier? I see that DecisionTreeClassifier accepts criterion='entropy', which means that it must be using information gain as a criterion for splitting the decision tree. What I need is the information gain for each feature at the root level, when it is about to split the root node.

Response: You can only access the information gain (or gini impurity) for a feature that has been used as a split node. The attribute DecisionTreeClassifier.tree_.best_error[i] holds the entropy of the i-th node splitting on feature DecisionTreeClassifier.tree_.feature[i]. If you want the entropy of all examples that reach the i-th node look at DecisionTreeClassifier.tree_.init_error[i]. For more information see the documentation here: https://github.com/scikit-learn/scikit-learn/blob/dacfd8bd5d943cb899ed8cd423aaf11b4f27c186/sklearn/tree/_tree.pyx#L64 If you want to access the entropy for each feature (at a certain split node) - you need to modify the function find_best_split https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/tree/_tree.pyx#L713


Question:
scikit-learn & statsmodels - which R-squared is correct? I'd like to choose the best algorithm for future. I found some solutions, but I didn't understand which R-Squared value is correct. For this, I divided my data into two as test and training, and I printed two different R squared values below.
```
import statsmodels.api as sm
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score

lineer = LinearRegression()
lineer.fit(x_train,y_train)
lineerPredict = lineer.predict(x_test)

scoreLineer = r2_score(y_test, lineerPredict)  # First R-Squared

model = sm.OLS(lineerPredict, y_test)
print(model.fit().summary()) # Second R-Squared
```
First R-Squared result is -4.28. Second R-Squared result is 0.84 But I didn't understand which value is correct.

Response:
Arguably, the real challenge in such cases is to be sure that you compare apples to apples. And in your case, it seems that you don't. Our best friend is always the relevant documentation, combined with simple experiments. So... Although scikit-learn's LinearRegression() (i.e. your 1st R-squared) is fitted by default with fit_intercept=True (docs), this is not the case with statsmodels' OLS (your 2nd R-squared); quoting from the docs: An intercept is not included by default and should be added by the user. See statsmodels.tools.add_constant. Keeping this important detail in mind, let's run some simple experiments with dummy data:
```
import numpy as np
import statsmodels.api as sm
from sklearn.metrics import r2_score
from sklearn.linear_model import LinearRegression

# dummy data:
y = np.array([1,3,4,5,2,3,4])
X = np.array(range(1,8)).reshape(-1,1) # reshape to column

# scikit-learn:
lr = LinearRegression()
lr.fit(X,y)
# LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
#     normalize=False)

lr.score(X,y)
# 0.16118421052631582

y_pred=lr.predict(X)
r2_score(y, y_pred)
# 0.16118421052631582

# statsmodels
# first artificially add intercept to X, as advised in the docs:
X_ = sm.add_constant(X)

model = sm.OLS(y,X_) # X_ here
results = model.fit()
results.rsquared
# 0.16118421052631593
```
For all practical purposes, these two values of R-squared produced by scikit-learn and statsmodels are identical. Let's go a step further, and try a scikit-learn model without intercept, but where we use the artificially "intercepted" data X_ we have already built for use with statsmodels:
```
lr2 = LinearRegression(fit_intercept=False)
lr2.fit(X_,y) # X_ here
# LinearRegression(copy_X=True, fit_intercept=False, n_jobs=None,
#         normalize=False)

lr2.score(X_, y)
# 0.16118421052631593

y_pred2 = lr2.predict(X_)
r2_score(y, y_pred2)
# 0.16118421052631593
```
Again, the R-squared is identical with the previous values. So, what happens when we "accidentally" forget to account for the fact that statsmodels OLS is fitted without an intercept? Let's see:
```
model3 = sm.OLS(y,X) # X here, i.e. no intercept
results3 = model2.fit()
results3.rsquared
# 0.8058035714285714
```
Well, an R-squared of 0.80 is indeed very far from the one of 0.16 returned by a model with an intercept, and arguably this is exactly what has happened in your case. So far so good, and I could easily finish the answer here; but there is indeed a point where this harmonious world breaks down: let's see what happens when we fit both models without intercept and with the initial data X where we have not artificially added any interception. We have already fitted the OLS model above, and got an R-squared of 0.80; what about a similar model from scikit-learn?
```
# scikit-learn
lr3 = LinearRegression(fit_intercept=False)
lr3.fit(X,y) # X here
lr3.score(X,y)
# -0.4309210526315792

y_pred3 = lr3.predict(X)
r2_score(y, y_pred3)
# -0.4309210526315792
```
Ooops...! What the heck?? It seems that scikit-earn, when computes the r2_score, always assumes an intercept, either explicitly in the model (fit_intercept=True) or implicitly in the data (the way we have produced X_ from X above, using statsmodels' add_constant); digging a little online reveals a Github thread (closed without a remedy) where it is confirmed that the situation is indeed like that. [UPDATE Dec 2021: for a more detailed & in-depth investigation and explanation of why the two scores are different in this particular case (i.e. both models fitted without an intercept), see this great answer by Flavia] Let me clarify that the discrepancy I have described above has nothing to do with your issue: in your case, the real issue is that you are actually comparing apples (a model with intercept) with oranges (a model without intercept). So, why scikit-learn not only fails in such an (admittedly edge) case, but even when the fact emerges in a Github issue it is actually treated with indifference? (Notice also that the scikit-learn core developer who replies in the above thread casually admits that "I'm not super familiar with stats"...). The answer goes a little beyond coding issues, such as the ones SO is mainly about, but it may be worth elaborating a little here. Arguably, the reason is that the whole R-squared concept comes in fact directly from the world of statistics, where the emphasis is on interpretative models, and it has little use in machine learning contexts, where the emphasis is clearly on predictive models; at least AFAIK, and beyond some very introductory courses, I have never (I mean never...) seen a predictive modeling problem where the R-squared is used for any kind of performance assessment; neither it's an accident that popular machine learning introductions, such as Andrew Ng's Machine Learning at Coursera, do not even bother to mention it. And, as noted in the Github thread above (emphasis added): In particular when using a test set, it's a bit unclear to me what the R^2 means. with which I certainly concur. As for the edge case discussed above (to include or not an intercept term?), I suspect it would sound really irrelevant to modern deep learning practitioners, where the equivalent of an intercept (bias parameters) is always included by default in neural network models... See the accepted (and highly upvoted) answer in the Cross Validated question Difference between statsmodel OLS and scikit linear regression for a more detailed discussion along these last lines. The discussion (and links) in Is R-squared Useless?, triggered by some relevant (negative) remarks by the great statistician Cosma Shalizi, is also enlightening and highly recommended.


Question:
Can someone explain to me how MinMaxScaler() works? [closed] Closed. This question needs to be more focused. It is not currently accepting answers. Want to improve this question? Guide the asker to update the question so it focuses on a single, specific problem. Narrowing the question will help others answer the question concisely. You may edit the question if you feel you can improve it yourself. If edited, the question will be reviewed and might be reopened. Closed 5 years ago. The community reviewed whether to reopen this question 3 years ago and left it closed: Original close reason(s) were not resolved Improve this question Why we are using the MinMaxScaler() and what does it do?
```
scaler = MinMaxScaler()
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
model = LogisticRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
```

Response: Core of the method A way to normalize the input features/variables is the Min-Max scaler. By doing so, all features will be transformed into the range [0,1] meaning that the minimum and maximum value of a feature/variable is going to be 0 and 1, respectively. Why to normalize prior to model fitting? The main idea behind normalization/standardization is always the same. Variables that are measured at different scales do not contribute equally to the model fitting & model learned function and might end up creating a bias. Thus, to deal with this potential problem feature-wise normalization such as MinMax Scaling is usually used prior to model fitting. More here: https://towardsdatascience.com/everything-you-need-to-know-about-min-max-normalization-in-python-b79592732b79


Question:
Simple prediction using linear regression with python
```
data2 = pd.DataFrame(data1['kwh'])
data2
                          kwh
date
2012-04-12 14:56:50     1.256400
2012-04-12 15:11:55     1.430750
2012-04-12 15:27:01     1.369910
2012-04-12 15:42:06     1.359350
2012-04-12 15:57:10     1.305680
2012-04-12 16:12:10     1.287750
2012-04-12 16:27:14     1.245970
2012-04-12 16:42:19     1.282280
2012-04-12 16:57:24     1.365710
2012-04-12 17:12:28     1.320130
2012-04-12 17:27:33     1.354890
2012-04-12 17:42:37     1.343680
2012-04-12 17:57:41     1.314220
2012-04-12 18:12:44     1.311970
2012-04-12 18:27:46     1.338980
2012-04-12 18:42:51     1.357370
2012-04-12 18:57:54     1.328700
2012-04-12 19:12:58     1.308200
2012-04-12 19:28:01     1.341770
2012-04-12 19:43:04     1.278350
2012-04-12 19:58:07     1.253170
2012-04-12 20:13:10     1.420670
2012-04-12 20:28:15     1.292740
2012-04-12 20:43:15     1.322840
2012-04-12 20:58:18     1.247410
2012-04-12 21:13:20     0.568352
2012-04-12 21:28:22     0.317865
2012-04-12 21:43:24     0.233603
2012-04-12 21:58:27     0.229524
2012-04-12 22:13:29     0.236929
2012-04-12 22:28:34     0.233806
2012-04-12 22:43:38     0.235618
2012-04-12 22:58:43     0.229858
2012-04-12 23:13:43     0.235132
2012-04-12 23:28:46     0.231863
2012-04-12 23:43:55     0.237794
2012-04-12 23:59:00     0.229634
2012-04-13 00:14:02     0.234484
2012-04-13 00:29:05     0.234189
2012-04-13 00:44:09     0.237213
2012-04-13 00:59:09     0.230483
2012-04-13 01:14:10     0.234982
2012-04-13 01:29:11     0.237121
2012-04-13 01:44:16     0.230910
2012-04-13 01:59:22     0.238406
2012-04-13 02:14:21     0.250530
2012-04-13 02:29:24     0.283575
2012-04-13 02:44:24     0.302299
2012-04-13 02:59:25     0.322093
2012-04-13 03:14:30     0.327600
2012-04-13 03:29:31     0.324368
2012-04-13 03:44:31     0.301869
2012-04-13 03:59:42     0.322019
2012-04-13 04:14:43     0.325328
2012-04-13 04:29:43     0.306727
2012-04-13 04:44:46     0.299012
2012-04-13 04:59:47     0.303288
2012-04-13 05:14:48     0.326205
2012-04-13 05:29:49     0.344230
2012-04-13 05:44:50     0.353484
...

65701 rows × 1 columns
```
I have this dataframe with this index and 1 column.I want to do simple prediction using linear regression with sklearn.I'm very confused and I don't know how to set X and y(I want the x values to be the time and y values kwh...).I'm new to Python so every help is valuable.Thank you.

Response:
The first thing you have to do is split your data into two arrays, X and y. Each element of X will be a date, and the corresponding element of y will be the associated kwh. Once you have that, you will want to use sklearn.linear_model.LinearRegression to do the regression. The documentation is here. As for every sklearn model, there are two steps. First you must fit your data. Then, put the dates of which you want to predict the kwh in another array, X_predict, and predict the kwh using the predict method.
```
from sklearn.linear_model import LinearRegression

X = []  # put your dates in here
y = []  # put your kwh in here

model = LinearRegression()
model.fit(X, y)

X_predict = []  # put the dates of which you want to predict kwh here
y_predict = model.predict(X_predict)
```