Question:
Sklearn SGDClassifier partial fit I'm trying to use SGD to classify a large dataset. As the data is too large to fit into memory, I'd like to use the partial_fit method to train the classifier. I have selected a sample of the dataset (100,000 rows) that fits into memory to test fit vs. partial_fit:
```
from sklearn.linear_model import SGDClassifier

def batches(l, n):
    for i in xrange(0, len(l), n):
        yield l[i:i+n]

clf1 = SGDClassifier(shuffle=True, loss='log')
clf1.fit(X, Y)

clf2 = SGDClassifier(shuffle=True, loss='log')
n_iter = 60
for n in range(n_iter):
    for batch in batches(range(len(X)), 10000):
        clf2.partial_fit(X[batch[0]:batch[-1]+1], Y[batch[0]:batch[-1]+1], classes=numpy.unique(Y))
```
I then test both classifiers with an identical test set. In the first case I get an accuracy of 100%. As I understand it, SGD by default passes 5 times over the training data (n_iter = 5). In the second case, I have to pass 60 times over the data to reach the same accuracy. Why this difference (5 vs. 60)? Or am I doing something wrong?

Response:
I have finally found the answer. You need to shuffle the training data between each iteration, as setting shuffle=True when instantiating the model will NOT shuffle the data when using partial_fit (it only applies to fit). Note: it would have been helpful to find this information on the sklearn.linear_model.SGDClassifier page. The amended code reads as follows:
```
from sklearn.linear_model import SGDClassifier
import random
clf2 = SGDClassifier(loss='log') # shuffle=True is useless here
shuffledRange = range(len(X))
n_iter = 5
for n in range(n_iter):
    random.shuffle(shuffledRange)
    shuffledX = [X[i] for i in shuffledRange]
    shuffledY = [Y[i] for i in shuffledRange]
    for batch in batches(range(len(shuffledX)), 10000):
        clf2.partial_fit(shuffledX[batch[0]:batch[-1]+1], shuffledY[batch[0]:batch[-1]+1], classes=numpy.unique(Y))
```


Question: How does sklearn.svm.svc's function predict_proba() work internally? I am using sklearn.svm.svc from scikit-learn to do binary classification. I am using its predict_proba() function to get probability estimates. Can anyone tell me how predict_proba() internally calculates the probability?

Response:
Scikit-learn uses LibSVM internally, and this in turn uses Platt scaling, as detailed in this note by the LibSVM authors, to calibrate the SVM to produce probabilities in addition to class predictions. Platt scaling requires first training the SVM as usual, then optimizing parameter vectors A and B such that
```
P(y|X) = 1 / (1 + exp(A * f(X) + B))
```
where f(X) is the signed distance of a sample from the hyperplane (scikit-learn's decision_function method). You may recognize the logistic sigmoid in this definition, the same function that logistic regression and neural nets use for turning decision functions into probability estimates. Mind you: the B parameter, the "intercept" or "bias" or whatever you like to call it, can cause predictions based on probability estimates from this model to be inconsistent with the ones you get from the SVM decision function f. E.g. suppose that f(X) = 10, then the prediction for X is positive; but if B = -9.9 and A = 1, then P(y|X) = .475. I'm pulling these numbers out of thin air, but you've noticed that this can occur in practice. Effectively, Platt scaling trains a probability model on top of the SVM's outputs under a cross-entropy loss function. To prevent this model from overfitting, it uses an internal five-fold cross validation, meaning that training SVMs with probability=True can be quite a lot more expensive than a vanilla, non-probabilistic SVM.


Question:
Sklearn StratifiedKFold: ValueError: Supported target types are: ('binary', 'multiclass'). Got 'multilabel-indicator' instead Working with Sklearn stratified kfold split, and when I attempt to split using multi-class, I received on error (see below). When I tried and split using binary, it works no problem.
```python
num_classes = len(np.unique(y_train))
y_train_categorical = keras.utils.to_categorical(y_train, num_classes)
kf=StratifiedKFold(n_splits=5, shuffle=True, random_state=999)

# splitting data into different folds
for i, (train_index, val_index) in enumerate(kf.split(x_train, y_train_categorical)):
    x_train_kf, x_val_kf = x_train[train_index], x_train[val_index]
    y_train_kf, y_val_kf = y_train[train_index], y_train[val_index]

ValueError: Supported target types are: ('binary', 'multiclass'). Got 'multilabel-indicator' instead.
```

Response: keras.utils.to_categorical produces a one-hot encoded class vector, i.e. the multilabel-indicator mentioned in the error message. StratifiedKFold is not designed to work with such input; from the split method docs: split(X, y, groups=None) [...] y : array-like, shape (n_samples,) The target variable for supervised learning problems. Stratification is done based on the y labels. i.e. your y must be a 1-D array of your class labels. Essentially, what you have to do is simply to invert the order of the operations: split first (using your intial y_train), and convert to_categorical afterwards.


Question:
Pass a dict to scikit learn estimator I am trying to pass model parameters as a dict to a Scikit-learn estimator and am having no luck. It just seems to nest my dict into one of the parameters. For instance:
```
params = {
 'copy_X': True,
 'fit_intercept': False,
 'normalize': True
}

lr = LinearRegression(params)
```
Gives me:
```
LinearRegression(copy_X=True,
         fit_intercept={'copy_X': True, 'fit_intercept': False,'normalize': True},
     normalize=False)
```
Additionally, I created a function to iterate over the dict and can create a string like:
```
'copy_X=True, fit_intercept=True, normalize=False'
```
This was equally as unsuccessful. Anyone have any advice here? The only restriction I have is the data will be coming to me as a dict (well actually a json object being loaded with json.uploads). Thanks.

Response:
The best solution to initialise your estimator with the right parameters would be to unpack your dictionary:
```
lr = LinearRegression(**params)
```
If for some reason you need to set some parameters afterwards, you could use:
```
lr.set_params(**params)
```
This has an advantage over using setattr in that it allows Scikit learn to perform some validation checks on the parameters.


Question:
Can I use CountVectorizer in scikit-learn to count frequency of documents that were not used to extract the tokens? I have been working with the CountVectorizer class in scikit-learn. I understand that if used in the manner shown below, the final output will consist of an array containing counts of features, or tokens. These tokens are extracted from a set of keywords, i.e.
```
tags = [
  "python, tools",
  "linux, tools, ubuntu",
  "distributed systems, linux, networking, tools",
]
```
The next step is:
```
from sklearn.feature_extraction.text import CountVectorizer
vec = CountVectorizer(tokenizer=tokenize)
data = vec.fit_transform(tags).toarray()
print data
```
Where we get
```
[[0 0 0 1 1 0]
 [0 1 0 0 1 1]
 [1 1 1 0 1 0]]
```
This is fine, but my situation is just a little bit different. I want to extract the features the same way as above, but I don't want the rows in data to be the same documents that the features were extracted from. In other words, how can I get counts of another set of documents, say,
```
list_of_new_documents = [
  ["python, chicken"],
  ["linux, cow, ubuntu"],
  ["machine learning, bird, fish, pig"]
]
```
And get:
```
[[0 0 0 1 0 0]
 [0 1 0 0 0 1]
 [0 0 0 0 0 0]]
```
I have read the documentation for the CountVectorizer class, and came across the vocabulary argument, which is a mapping of terms to feature indices. I can't seem to get this argument to help me, however. Any advice is appreciated. PS: all credit due to Matthias Friedrich's Blog for the example I used above.

Response:
You're right that vocabulary is what you want. It works like this:
```

```python
cv = sklearn.feature_extraction.text.CountVectorizer(vocabulary=['hot', 'cold', 'old'])
```

```python
cv.fit_transform(['pease porridge hot', 'pease porridge cold', 'pease porridge in the pot', 'nine days old']).toarray()
#Output
#array([[1, 0, 0],
#       [0, 1, 0],
#       [0, 0, 0],
#       [0, 0, 1]], dtype=int64)
#```
#So you pass it a dict with your desired features as the keys. If you used CountVectorizer on one set of documents and then you want to use the set of features from those documents for a new set, use the vocabulary_ attribute of your original CountVectorizer and pass it to the new one. So in your example, you could do
#```
#newVec = CountVectorizer(vocabulary=vec.vocabulary_)
#```
#to create a new tokenizer using the vocabulary from your first one.
```


Question:
Understanding the `ngram_range` argument in a CountVectorizer in sklearn I'm a little confused about how to use ngrams in the scikit-learn library in Python, specifically, how the ngram_range argument works in a CountVectorizer. Running this code:
```
from sklearn.feature_extraction.text import CountVectorizer
vocabulary = ['hi ', 'bye', 'run away']
cv = CountVectorizer(vocabulary=vocabulary, ngram_range=(1, 2))
print cv.vocabulary_
```
gives me:
```
{'hi ': 0, 'bye': 1, 'run away': 2}
```
Where I was under the (obviously mistaken) impression that I would get unigrams and bigrams, like this:
```
{'hi ': 0, 'bye': 1, 'run away': 2, 'run': 3, 'away': 4}
```
I am working with the documentation here: http://scikit-learn.org/stable/modules/feature_extraction.html Clearly there is something terribly wrong with my understanding of how to use ngrams. Perhaps the argument is having no effect or I have some conceptual issue with what an actual bigram is! I'm stumped. If anyone has a word of advice to throw my way, I'd be grateful. UPDATE: I have realized the folly of my ways. I was under the impression that the ngram_range would affect the vocabulary, not the corpus.

Response:
Setting the vocabulary explicitly means no vocabulary is learned from data. If you don't set it, you get:
```

```python
v = CountVectorizer(ngram_range=(1, 2))
```

```python
pprint(v.fit(["an apple a day keeps the doctor away"]).vocabulary_)
#Output
#{u'an': 0,
# u'an apple': 1,
# u'apple': 2,
# u'apple day': 3,
# u'away': 4,
# u'day': 5,
# u'day keeps': 6,
# u'doctor': 7,
# u'doctor away': 8,
# u'keeps': 9,
# u'keeps the': 10,
# u'the': 11,
# u'the doctor': 12}
#```
#An explicit vocabulary restricts the terms that will be extracted from text; the vocabulary is not changed:
#```
```

```python
v = CountVectorizer(ngram_range=(1, 2), vocabulary={"keeps", "keeps the"})
```

```python
v.fit_transform(["an apple a day keeps the doctor away"]).toarray()
#Output
#array([[1, 1]])  # unigram and bigram found
#```
#(Note that stopword filtering is applied before n-gram extraction, hence "apple day".)
```


Question:
What is "random-state" in sklearn.model_selection.train_test_split example? [duplicate] This question already has answers here: Random state (Pseudo-random number) in Scikit learn (8 answers) Closed 4 years ago. Can someone explain me what random_state means in below example?
```
import numpy as np
from sklearn.model_selection import train_test_split
X, y = np.arange(10).reshape((5, 2)), range(5)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=42)
```
Why is it hard coded to 42?

Response: Isn't that obvious? 42 is the Answer to the Ultimate Question of Life, the Universe, and Everything. On a serious note, random_state simply sets a seed to the random generator, so that your train-test splits are always deterministic. If you don't set a seed, it is different each time. Relevant documentation: random_state : int, RandomState instance or None, optional (default=None) If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.


Question:
sklearn doesn't have attribute 'datasets' I have started using sckikit-learn for my work. So I was going through the tutorial which gives standard procedure to load some datasets:
```
$ python
>>> from sklearn import datasets
>>> iris = datasets.load_iris()
>>> digits = datasets.load_digits()
```
However, for my convenience, I tried loading the data in the following way:
```

```python
import sklearn
```

```python
iris = sklearn.datasets.load_iris()
```
However, this throws following error:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-db77d2036db5> in <module>()
----> 1 iris = sklearn.datasets.load_iris()

AttributeError: 'module' object has no attribute 'datasets'
```
However, if I use the apparently similar method:
```
```

```python
from sklearn import datasets
```

```python
iris = datasets.load_iris()
```
It works without problem. In fact the following also works:
```
```

```python
iris = sklearn.datasets.load_iris()
```
I am completely confused about this. Am I missing something very trivial? What is the difference between the two approaches?
```

Response:
sklearn is a package. This answer said it very succinctly: when you import a package, only variables/functions/classes in the __init__.py file of that package are directly visible, not sub-packages or modules. datasets is a sub-package of sklearn. This is why this happens:
```

```python
import sklearn
```

```python
sklearn.datasets
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-325a2bfc35d0> in <module>()
----> 1 sklearn.datasets

AttributeError: module 'sklearn' has no attribute 'datasets'
```
However, the reason why this works:
```
```

```python
from sklearn import datasets
```

```python
sklearn.datasets
#Output
#<module 'sklearn.datasets' from '/home/ethan/.virtualenvs/test3/lib/python3.5/site-packages/sklearn/datasets/__init__.py'>
#```
#is that when you load the sub-package datasets by doing from sklearn import datasets it is automatically added to the namespace of the package sklearn. This is one of the lesser-known "traps" of the Python import system. Also, note that if you look at the __init__.py for sklearn you will see 'datasets' as a member of __all__, but this only allows you to do:
#```
```

```python
from sklearn import *
```

```python
datasets
#Output
#<module 'sklearn.datasets' from '/home/ethan/.virtualenvs/test3/lib/python3.5/site-packages/sklearn/datasets/__init__.py'>
#```
#One last point to note is that if you inspect either sklearn or datasets you will see that, although they are packages, their type is module. This is because all packages are considered modules - however, not all modules are packages.
```


Question:
sklearn : TFIDF Transformer : How to get tf-idf values of given words in document I used sklearn for calculating TFIDF (Term frequency inverse document frequency) values for documents using command as :
```
from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(documents)
from sklearn.feature_extraction.text import TfidfTransformer
tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)
X_train_tf = tf_transformer.transform(X_train_counts)
```
X_train_tf is a scipy.sparse matrix of shape (2257, 35788). How can I get TF-IDF for words in a particular document? More specific, how to get words with maximum TF-IDF values in a given document?

Response:
You can use TfidfVectorizer from sklean
```
from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np
from scipy.sparse.csr import csr_matrix #need this if you want to save tfidf_matrix

tf = TfidfVectorizer(input='filename', analyzer='word', ngram_range=(1,6),
                     min_df = 0, stop_words = 'english', sublinear_tf=True)
tfidf_matrix =  tf.fit_transform(corpus)
```
The above tfidf_matix has the TF-IDF values of all the documents in the corpus. This is a big sparse matrix. Now,
```
feature_names = tf.get_feature_names()
```
this gives you the list of all the tokens or n-grams or words. For the first document in your corpus,
```
doc = 0
feature_index = tfidf_matrix[doc,:].nonzero()[1]
tfidf_scores = zip(feature_index, [tfidf_matrix[doc, x] for x in feature_index])
```
Lets print them,
```
for w, s in [(feature_names[i], s) for (i, s) in tfidf_scores]:
  print w, s
```


Question:
How to see top n entries of term-document matrix after tfidf in scikit-learn I am new to scikit-learn, and I was using TfidfVectorizer to find the tfidf values of terms in a set of documents. I used the following code to obtain the same.
```
vectorizer = TfidfVectorizer(stop_words=u'english',ngram_range=(1,5),lowercase=True)
X = vectorizer.fit_transform(lectures)
```
Now If I print X, I am able to see all the entries in matrix, but how can I find top n entries based on tfidf score. In addition to that is there any method that will help me to find top n entries based on tfidf score per ngram i.e. top entries among unigram,bigram,trigram and so on?

Response:
Since version 0.15, the global term weighting of the features learnt by a TfidfVectorizer can be accessed through the attribute idf_, which will return an array of length equal to the feature dimension. Sort the features by this weighting to get the top weighted features:
```
from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np

lectures = ["this is some food", "this is some drink"]
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(lectures)
indices = np.argsort(vectorizer.idf_)[::-1]
features = vectorizer.get_feature_names()
top_n = 2
top_features = [features[i] for i in indices[:top_n]]
print top_features
```
Output:
```
[u'food', u'drink']
```
The second problem of getting the top features by ngram can be done using the same idea, with some extra steps of splitting the features into different groups:
```
from sklearn.feature_extraction.text import TfidfVectorizer
from collections import defaultdict

lectures = ["this is some food", "this is some drink"]
vectorizer = TfidfVectorizer(ngram_range=(1,2))
X = vectorizer.fit_transform(lectures)
features_by_gram = defaultdict(list)
for f, w in zip(vectorizer.get_feature_names(), vectorizer.idf_):
    features_by_gram[len(f.split(' '))].append((f, w))
top_n = 2
for gram, features in features_by_gram.iteritems():
    top_features = sorted(features, key=lambda x: x[1], reverse=True)[:top_n]
    top_features = [f[0] for f in top_features]
    print '{}-gram top:'.format(gram), top_features
```
Output:
```
1-gram top: [u'drink', u'food']
2-gram top: [u'some drink', u'some food']
```


Question: Understanding max_features parameter in RandomForestRegressor While constructing each tree in the random forest using bootstrapped samples, for each terminal node, we select m variables at random from p variables to find the best split (p is the total number of features in your data). My questions (for RandomForestRegressor) are: 1) What does max_features correspond to (m or p or something else)? 2) Are m variables selected at random from max_features variables (what is the value of m)? 3) If max_features corresponds to m, then why would I want to set it equal to p for regression (the default)? Where is the randomness with this setting (i.e., how is it different from bagging)? Thanks.

Response: Straight from the documentation: [max_features] is the size of the random subsets of features to consider when splitting a node. So max_features is what you call m. When max_features="auto", m = p and no feature subset selection is performed in the trees, so the "random forest" is actually a bagged ensemble of ordinary regression trees. The docs go on to say that Empirical good default values are max_features=n_features for regression problems, and max_features=sqrt(n_features) for classification tasks By setting max_features differently, you'll get a "true" random forest.


Question:
sklearn metrics for multiclass classification I have performed GaussianNB classification using sklearn. I tried to calculate the metrics using the following code:
```
print accuracy_score(y_test, y_pred)
print precision_score(y_test, y_pred)
```
Accuracy score is working correctly but precision score calculation is showing error as: ValueError: Target is multiclass but average='binary'. Please choose another average setting. As target is multiclass, can i have the metric scores of precision, recall etc.?

Response: The function call precision_score(y_test, y_pred) is equivalent to precision_score(y_test, y_pred, pos_label=1, average='binary'). The documentation (http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html) tells us: 'binary': Only report results for the class specified by pos_label. This is applicable only if targets (y_{true,pred}) are binary. So the problem is that your labels are not binary, but probably one-hot encoded. Fortunately, there are other options which should work with your data: precision_score(y_test, y_pred, average=None) will return the precision scores for each class, while precision_score(y_test, y_pred, average='micro') will return the total ratio of tp/(tp + fp) The pos_label argument will be ignored if you choose another average option than binary.


Question:
Multilabel-indicator is not supported for confusion matrix multilabel-indicator is not supported is the error message I get, when trying to run: confusion_matrix(y_test, predictions) y_test is a DataFrame which is of shape:
```
Horse | Dog | Cat
1       0     0
0       1     0
0       1     0
...     ...   ...
```
predictions is a numpy array:
```
[[1, 0, 0],
 [0, 1, 0],
 [0, 1, 0]]
```
I've searched a bit for the error message, but haven't really found something I could apply. Any hints?

Response:
No, your input to confusion_matrix must be a list of predictions, not OHEs (one hot encodings). Call argmax on your y_test and y_pred, and you should get what you expect.
```
confusion_matrix(
    y_test.values.argmax(axis=1), predictions.argmax(axis=1))

array([[1, 0],
       [0, 2]])
```


Question: What is the difference between SVC and SVM in scikit-learn? From the documentation scikit-learn implements SVC, NuSVC and LinearSVC which are classes capable of performing multi-class classification on a dataset. By the other hand I also read about that scikit learn also uses libsvm for support vector machine algorithm. I'm a bit confused about what's the difference between SVC and libsvm versions, by now I guess the difference is that SVC is the support vector machine algorithm fot the multiclass problem and libsvm is for the binary class problem. Could anybody help me to understad the difference between this?.

Response:
They are just different implementations of the same algorithm. The SVM module (SVC, NuSVC, etc) is a wrapper around the libsvm library and supports different kernels while LinearSVC is based on liblinear and only supports a linear kernel. So:
```
SVC(kernel = 'linear')
```
is in theory "equivalent" to:
```
LinearSVC()
```
Because the implementations are different in practice you will get different results, the most important ones being that LinearSVC only supports a linear kernel, is faster and can scale a lot better.


Question:
How is scikit-learn cross_val_predict accuracy score calculated? Does the cross_val_predict (see doc, v0.18) with k-fold method as shown in the code below calculate accuracy for each fold and average them finally or not?
```
cv = KFold(len(labels), n_folds=20)
clf = SVC()
ypred = cross_val_predict(clf, td, labels, cv=cv)
accuracy = accuracy_score(labels, ypred)
print accuracy
```

Response:
No, it does not! According to cross validation doc page, cross_val_predict does not return any scores but only the labels based on a certain strategy which is described here: The function cross_val_predict has a similar interface to cross_val_score, but returns, for each element in the input, the prediction that was obtained for that element when it was in the test set. Only cross-validation strategies that assign all elements to a test set exactly once can be used (otherwise, an exception is raised). And therefore by calling accuracy_score(labels, ypred) you are just calculating accuracy scores of labels predicted by aforementioned particular strategy compared to the true labels. This again is specified in the same documentation page: These prediction can then be used to evaluate the classifier:
```
predicted = cross_val_predict(clf, iris.data, iris.target, cv=10)
metrics.accuracy_score(iris.target, predicted)
```
Note that the result of this computation may be slightly different from those obtained using cross_val_score as the elements are grouped in different ways. If you need accuracy scores of different folds you should try:
```

```python
scores = cross_val_score(clf, X, y, cv=cv)
```

```python
scores
#Output
#array([ 0.96...,  1.  ...,  0.96...,  0.96...,  1.        ])
#```
#and then for the mean accuracy of all folds use scores.mean():
#```
```

```python
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
#Output
#Accuracy: 0.98 (+/- 0.03)
#```
#How to calculate Cohen kappa coefficient and confusion matrix for each fold? For calculating Cohen Kappa coefficient and confusion matrix I assumed you mean kappa coefficient and confusion matrix between true labels and each fold's predicted labels:
#```
#from sklearn.model_selection import KFold
#from sklearn.svm.classes import SVC
#from sklearn.metrics.classification import cohen_kappa_score
#from sklearn.metrics import confusion_matrix
```

cv = KFold(len(labels), n_folds=20)
clf = SVC()
for train_index, test_index in cv.split(X):
    clf.fit(X[train_index], labels[train_index])
    ypred = clf.predict(X[test_index])
    kappa_score = cohen_kappa_score(labels[test_index], ypred)
    confusion_matrix = confusion_matrix(labels[test_index], ypred)
```
What does cross_val_predict return? It uses KFold to split the data to k parts and then for i=1..k iterations: takes i'th part as the test data and all other parts as training data trains the model with training data (all parts except i'th) then by using this trained model, predicts labels for i'th part (test data) In each iteration, label of i'th part of data gets predicted. In the end cross_val_predict merges all partially predicted labels and returns them as the final result. This code shows this process step by step:
```
X = np.array([[0], [1], [2], [3], [4], [5]])
labels = np.array(['a', 'a', 'a', 'b', 'b', 'b'])

cv = KFold(len(labels), n_folds=3)
clf = SVC()
ypred_all = np.chararray((labels.shape))
i = 1
for train_index, test_index in cv.split(X):
    print("iteration", i, ":")
    print("train indices:", train_index)
    print("train data:", X[train_index])
    print("test indices:", test_index)
    print("test data:", X[test_index])
    clf.fit(X[train_index], labels[train_index])
    ypred = clf.predict(X[test_index])
    print("predicted labels for data of indices", test_index, "are:", ypred)
    ypred_all[test_index] = ypred
    print("merged predicted labels:", ypred_all)
    i = i+1
    print("=====================================")
y_cross_val_predict = cross_val_predict(clf, X, labels, cv=cv)
print("predicted labels by cross_val_predict:", y_cross_val_predict)
```
The result is:
```
iteration 1 :
train indices: [2 3 4 5]
train data: [[2] [3] [4] [5]]
test indices: [0 1]
test data: [[0] [1]]
predicted labels for data of indices [0 1] are: ['b' 'b']
merged predicted labels: ['b' 'b' '' '' '' '']
=====================================
iteration 2 :
train indices: [0 1 4 5]
train data: [[0] [1] [4] [5]]
test indices: [2 3]
test data: [[2] [3]]
predicted labels for data of indices [2 3] are: ['a' 'b']
merged predicted labels: ['b' 'b' 'a' 'b' '' '']
=====================================
iteration 3 :
train indices: [0 1 2 3]
train data: [[0] [1] [2] [3]]
test indices: [4 5]
test data: [[4] [5]]
predicted labels for data of indices [4 5] are: ['a' 'a']
merged predicted labels: ['b' 'b' 'a' 'b' 'a' 'a']
=====================================
predicted labels by cross_val_predict: ['b' 'b' 'a' 'b' 'a' 'a']
```


Question: Adding words to scikit-learn's CountVectorizer's stop list Scikit-learn's CountVectorizer class lets you pass a string 'english' to the argument stop_words. I want to add some things to this predefined list. Can anyone tell me how to do this?

Response:
According to the source code for sklearn.feature_extraction.text, the full list (actually a frozenset, from stop_words) of ENGLISH_STOP_WORDS is exposed through __all__. Therefore if you want to use that list plus some more items, you could do something like:
```
from sklearn.feature_extraction import text

stop_words = text.ENGLISH_STOP_WORDS.union(my_additional_stop_words)
```
(where my_additional_stop_words is any sequence of strings) and use the result as the stop_words argument. This input to CountVectorizer.__init__ is parsed by _check_stop_list, which will pass the new frozenset straight through.


Question: What is the difference between xgb.train and xgb.XGBRegressor (or xgb.XGBClassifier)? I already know "xgboost.XGBRegressor is a Scikit-Learn Wrapper interface for XGBoost." But do they have any other difference?

Response:
xgboost.train is the low-level API to train the model via gradient boosting method. xgboost.XGBRegressor and xgboost.XGBClassifier are the wrappers (Scikit-Learn-like wrappers, as they call it) that prepare the DMatrix and pass in the corresponding objective function and parameters. In the end, the fit call simply boils down to:
```py
self._Booster = train(params, dmatrix,
                      self.n_estimators, evals=evals,
                      early_stopping_rounds=early_stopping_rounds,
                      evals_result=evals_result, obj=obj, feval=feval,
                      verbose_eval=verbose)
```
This means that everything that can be done with XGBRegressor and XGBClassifier is doable via underlying xgboost.train function. The other way around it's obviously not true, for instance, some useful parameters of xgboost.train are not supported in XGBModel API. The list of notable differences includes: xgboost.train allows to set the callbacks applied at end of each iteration. xgboost.train allows training continuation via xgb_model parameter. xgboost.train allows not only minization of the eval function, but maximization as well.


Question:
Sklearn set_params takes exactly 1 argument? I'm trying to use SkLearn Bayes classification.
```
gnb = GaussianNB()
 gnb.set_params('sigma__0.2')
 gnb.fit(np.transpose([xn, yn]), y)
```
But I get:
```
set_params() takes exactly 1 argument (2 given)
```
now I try to use this code:
```
gnb = GaussianNB()
arr = np.zeros((len(labs),len(y)))
arr.fill(sigma)
gnb.set_params(sigma_ = arr)
```
And get:
```
ValueError: Invalid parameter sigma_ for estimator GaussianNB
```
Is it wrong parameter name or value?

Response:
I just stumbled upon this, so here is a solution for multiple arguments from a dictionary:
```
from sklearn import svm
params_svm = {"kernel":"rbf", "C":0.1, "gamma":0.1, "class_weight":"auto"}
clf = svm.SVC()
clf.set_params(**params_svm)
```


Question: What's the best way to test whether an sklearn model has been fitted? What's the most elegant way to check whether an sklearn model has been fitted? i.e. whether its fit() function has been called after it was instantiated, or not.

Response:
You can do something like:
```
from sklearn.exceptions import NotFittedError

for model in models:
    try:
        model.predict(some_test_data)
    except NotFittedError as e:
        print(repr(e))
```
Ideally you would check the results of model.predict against expected results but if all you want to know if wether the model is fitted or not that should suffice. Update: Some commenters have suggested using check_is_fitted. I consider check_is_fitted an internal method. Most algorithms will call check_is_fitted inside their predict method which in turn might raise NotFittedError if needed. The problem with using check_is_fitted directly is that it is model specific, i.e. you need to know which members to check depending on your algorithm. For example:
```
╔════════════════╦════════════════════════════════════════════╗
║ Tree models    ║ check_is_fitted(self, 'tree_')             ║
║ Linear models  ║ check_is_fitted(self, 'coefs_')            ║
║ KMeans         ║ check_is_fitted(self, 'cluster_centers_')  ║
║ SVM            ║ check_is_fitted(self, 'support_')          ║
╚════════════════╩════════════════════════════════════════════╝
```
and so on. So in general I would recommend calling model.predict() and letting the specific algorithm handle the best way to check whether it is already fitted or not.


Question:
Scikit Learn: Logistic Regression model coefficients: Clarification I need to know how to return the logistic regression coefficients in such a manner that I can generate the predicted probabilities myself. My code looks like this:
```
lr = LogisticRegression()
lr.fit(training_data, binary_labels)

# Generate probabities automatically
predicted_probs = lr.predict_proba(binary_labels)
```
I had assumed the lr.coeff_ values would follow typical logistic regression, so that I could return the predicted probabilities like this:
```
sigmoid( dot([val1, val2, offset], lr.coef_.T) )
```
But this is not the appropriate formulation. Does anyone have the proper format for generating predicted probabilities from Scikit Learn LogisticRegression? Thanks!

Response:
take a look at the documentations (http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html), offset coefficient isn't stored by lr.coef_ coef_ array, shape = [n_classes-1, n_features] Coefficient of the features in the decision function. coef_ is readonly property derived from raw_coef_ that follows the internal memory layout of liblinear. intercept_ array, shape = [n_classes-1] Intercept (a.k.a. bias) added to the decision function. It is available only when parameter intercept is set to True. try:
```
sigmoid( dot([val1, val2], lr.coef_) + lr.intercept_ )
```


Question:
How do I get word frequency in a corpus using Scikit Learn CountVectorizer? I'm trying to compute a simple word frequency using scikit-learn's CountVectorizer.
```
import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer

texts=["dog cat fish","dog cat cat","fish bird","bird"]
cv = CountVectorizer()
cv_fit=cv.fit_transform(texts)

print cv.vocabulary_
{u'bird': 0, u'cat': 1, u'dog': 2, u'fish': 3}
```
I was expecting it to return {u'bird': 2, u'cat': 3, u'dog': 2, u'fish': 2}.

Response:
cv.vocabulary_ in this instance is a dict, where the keys are the words (features) that you've found and the values are indices, which is why they're 0, 1, 2, 3. It's just bad luck that it looked similar to your counts :) You need to work with the cv_fit object to get the counts
```
from sklearn.feature_extraction.text import CountVectorizer

texts = ["dog cat fish", "dog cat cat", "fish bird", "bird"]
cv = CountVectorizer()
cv_fit = cv.fit_transform(texts)

print(cv.get_feature_names())
print(cv_fit.toarray())
# ["bird", "cat", "dog", "fish"]
# [[0 1 1 1]
#  [0 2 1 0]
#  [1 0 0 1]
#  [1 0 0 0]]
```
Each row in the array is one of your original documents (strings), each column is a feature (word), and the element is the count for that particular word and document. You can see that if you sum each column you'll get the correct number
```
print(cv_fit.toarray().sum(axis=0))
# [2 3 2 2]
```
Honestly though, I'd suggest using collections.Counter or something from NLTK, unless you have some specific reason to use scikit-learn, as it'll be simpler.


Question:
Issue with OneHotEncoder for categorical features I want to encode 3 categorical features out of 10 features in my datasets. I use preprocessing from sklearn.preprocessing to do so as the following:
```
from sklearn import preprocessing
cat_features = ['color', 'director_name', 'actor_2_name']
enc = preprocessing.OneHotEncoder(categorical_features=cat_features)
enc.fit(dataset.values)
```
However, I couldn't proceed as I am getting this error:
```
array = np.array(array, dtype=dtype, order=order, copy=copy)
ValueError: could not convert string to float: PG
```
I am surprised why it is complaining about the string as it is supposed to convert it!! Am I missing something here?

Response:
If you read the docs for OneHotEncoder you'll see the input for fit is "Input array of type int". So you need to do two steps for your one hot encoded data
```
from sklearn import preprocessing
cat_features = ['color', 'director_name', 'actor_2_name']
enc = preprocessing.LabelEncoder()
enc.fit(cat_features)
new_cat_features = enc.transform(cat_features)
print new_cat_features # [1 2 0]
new_cat_features = new_cat_features.reshape(-1, 1) # Needs to be the correct shape
ohe = preprocessing.OneHotEncoder(sparse=False) #Easier to read
print ohe.fit_transform(new_cat_features)
```
Output:
```
[[ 0.  1.  0.]
 [ 0.  0.  1.]
 [ 1.  0.  0.]]
```
EDIT As of 0.20 this became a bit easier, not only because OneHotEncoder now handles strings nicely, but also because we can transform multiple columns easily using ColumnTransformer, see below for an example
```
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
import numpy as np

X = np.array([['apple', 'red', 1, 'round', 0],
              ['orange', 'orange', 2, 'round', 0.1],
              ['bannana', 'yellow', 2, 'long', 0],
              ['apple', 'green', 1, 'round', 0.2]])
ct = ColumnTransformer(
    [('oh_enc', OneHotEncoder(sparse=False), [0, 1, 3]),],  # the column numbers I want to apply this to
    remainder='passthrough'  # This leaves the rest of my columns in place
)
print(ct2.fit_transform(X)) # Notice the output is a string
```
Output:
```
[['1.0' '0.0' '0.0' '0.0' '0.0' '1.0' '0.0' '0.0' '1.0' '1' '0']
 ['0.0' '0.0' '1.0' '0.0' '1.0' '0.0' '0.0' '0.0' '1.0' '2' '0.1']
 ['0.0' '1.0' '0.0' '0.0' '0.0' '0.0' '1.0' '1.0' '0.0' '2' '0']
 ['1.0' '0.0' '0.0' '1.0' '0.0' '0.0' '0.0' '0.0' '1.0' '1' '0.2']]
```


Question: Fitting data vs. transforming data in scikit-learn In scikit-learn, all estimators have a fit() method, and depending on whether they are supervised or unsupervised, they also have a predict() or transform() method. I am in the process of writing a transformer for an unsupervised learning task and was wondering if there is a rule of thumb where to put which kind of learning logic. The official documentation is not very helpful in this regard: fit_transform(X, y=None, **fit_params) Fit to data, then transform it. In this context, what is meant by both fitting data and transforming data?

Response:
Fitting finds the internal parameters of a model that will be used to transform data. Transforming applies the parameters to data. You may fit a model to one set of data, and then transform it on a completely different set. For example, you fit a linear model to data to get a slope and intercept. Then you use those parameters to transform (i.e., map) new or existing values of x to y. fit_transform is just doing both steps to the same data. A scikit example: You fit data to find the principal components. Then you transform your data to see how it maps onto these components:
```
from sklearn.decomposition import PCA

pca = PCA(n_components=2)

X = [[1,2],[2,4],[1,3]]

pca.fit(X)

# This is the model to map data
pca.components_

array([[ 0.47185791,  0.88167459],
       [-0.88167459,  0.47185791]], dtype=float32)

# Now we actually map the data
pca.transform(X)

array([[-1.03896057, -0.17796634],
       [ 1.19624651, -0.11592512],
       [-0.15728599,  0.29389156]])

# Or we can do both "at once"
pca.fit_transform(X)

array([[-1.03896058, -0.1779664 ],
       [ 1.19624662, -0.11592512],
       [-0.15728603,  0.29389152]], dtype=float32)
```


Question: How do you access tree depth in Python's scikit-learn? I'm using scikit-learn to create a Random Forest. However, I want to find the individual depths of each tree. It seems like a simple attribute to have but according to the documentation, (http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) there is no way of accessing it. If this isn't possible, is there a way of accessing the tree depth from a Decision Tree model? Any help would be appreciated. Thank you.

Response:
Each instance of RandomForestClassifier has an estimators_ attribute, which is a list of DecisionTreeClassifier instances. The documentation shows that an instance of DecisionTreeClassifier has a tree_ attribute, which is an instance of the (undocumented, I believe) Tree class. Some exploration in the interpreter shows that each Tree instance has a max_depth parameter which appears to be what you're looking for -- again, it's undocumented. In any case, if forest is your instance of RandomForestClassifier, then:
```

```python
[estimator.tree_.max_depth for estimator in forest.estimators_]
#Output
#[9, 10, 9, 11, 9, 9, 11, 7, 13, 10]
#```
#should do the trick. Each estimator also has a get_depth() method than can be used to retrieve the same value with briefer syntax:
#```
```

```python
[estimator.get_depth() for estimator in forest.estimators_]
#Output
#[9, 10, 9, 11, 9, 9, 11, 7, 13, 10]
#```
#To avoid mixup, it should be noted that there is an attribute of each estimator (and not each estimator's tree_) called max depth which returns the setting of the parameter rather than the depth of the actual tree. How estimator.get_depth(), estimator.tree_.max_depth, and estimator.max_depth relate to each other is clarified in the example below:
#```
#from sklearn.datasets import load_iris
#from sklearn.ensemble import RandomForestClassifier
#clf = RandomForestClassifier(n_estimators=3, random_state=4, max_depth=6)
#iris = load_iris()
#clf.fit(iris['data'], iris['target'])
#[(est.get_depth(), est.tree_.max_depth, est.max_depth) for est in clf.estimators_]
#```
#Out:
#```
#[(6, 6, 6), (3, 3, 6), (4, 4, 6)]
#```
#Setting max depth to the default value None would allow the first tree to expand to depth 7 and the output would be:
#```
#[(7, 7, None), (3, 3, None), (4, 4, None)]
#```
```


Question: Spectral Clustering a graph in python I'd like to cluster a graph in python using spectral clustering. Spectral clustering is a more general technique which can be applied not only to graphs, but also images, or any sort of data, however, it's considered an exceptional graph clustering technique. Sadly, I can't find examples of spectral clustering graphs in python online. Scikit Learn has two spectral clustering methods documented: SpectralClustering and spectral_clustering which seem like they're not aliases. Both of those methods mention that they could be used on graphs, but do not offer specific instructions. Neither does the user guide. I've asked for such an example from the developers, but they're overworked and haven't gotten to it. A good network to document this against is the Karate Club Network. It's included as a method in networkx. I'd love some direction in how to go about this. If someone can help me figure it out, I can add the documentation to scikit learn. Notes: A question much like this one has already been asked on this site.

Response:
Without much experience with Spectral-clustering and just going by the docs (skip to the end for the results!): Code:
```
import numpy as np
import networkx as nx
from sklearn.cluster import SpectralClustering
from sklearn import metrics
np.random.seed(1)

# Get your mentioned graph
G = nx.karate_club_graph()

# Get ground-truth: club-labels -> transform to 0/1 np-array
#     (possible overcomplicated networkx usage here)
gt_dict = nx.get_node_attributes(G, 'club')
gt = [gt_dict[i] for i in G.nodes()]
gt = np.array([0 if i == 'Mr. Hi' else 1 for i in gt])

# Get adjacency-matrix as numpy-array
adj_mat = nx.to_numpy_matrix(G)

print('ground truth')
print(gt)

# Cluster
sc = SpectralClustering(2, affinity='precomputed', n_init=100)
sc.fit(adj_mat)

# Compare ground-truth and clustering-results
print('spectral clustering')
print(sc.labels_)
print('just for better-visualization: invert clusters (permutation)')
print(np.abs(sc.labels_ - 1))

# Calculate some clustering metrics
print(metrics.adjusted_rand_score(gt, sc.labels_))
print(metrics.adjusted_mutual_info_score(gt, sc.labels_))
```
Output:
```
ground truth
[0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 1 0 1 0 1 1 1 1 1 1 1 1 1 1 1 1]
spectral clustering
[1 1 0 1 1 1 1 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
just for better-visualization: invert clusters (permutation)
[0 0 1 0 0 0 0 1 1 1 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
0.204094758281
0.271689477828
```
The general idea: Introduction on the data and task from here: The nodes in the graph represent the 34 members in a college Karate club. (Zachary is a sociologist, and he was one of the members.) An edge between two nodes indicates that the two members spent significant time together outside normal club meetings. The dataset is interesting because while Zachary was collecting his data, there was a dispute in the Karate club, and it split into two factions: one led by “Mr. Hi”, and one led by “John A”. It turns out that using only the connectivity information (the edges), it is possible to recover the two factions. Using sklearn & spectral-clustering to tackle this: If affinity is the adjacency matrix of a graph, this method can be used to find normalized graph cuts. This describes normalized graph cuts as: Find two disjoint partitions A and B of the vertices V of a graph, so that A ∪ B = V and A ∩ B = ∅ Given a similarity measure w(i,j) between two vertices (e.g. identity when they are connected) a cut value (and its normalized version) is defined as: cut(A, B) = SUM u in A, v in B: w(u, v) ... we seek the minimization of disassociation between the groups A and B and the maximization of the association within each group Sounds alright. So we create the adjacency matrix (nx.to_numpy_matrix(G)) and set the param affinity to precomputed (as our adjancency-matrix is our precomputed similarity-measure). Alternatively, using precomputed, a user-provided affinity matrix can be used. Edit: While unfamiliar with this, i looked for parameters to tune and found assign_labels: The strategy to use to assign labels in the embedding space. There are two ways to assign labels after the laplacian embedding. k-means can be applied and is a popular choice. But it can also be sensitive to initialization. Discretization is another approach which is less sensitive to random initialization. So trying the less sensitive approach:
```
sc = SpectralClustering(2, affinity='precomputed', n_init=100, assign_labels='discretize')
```
Output:
```
ground truth
[0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 1 0 1 0 1 1 1 1 1 1 1 1 1 1 1 1]
spectral clustering
[0 0 1 0 0 0 0 0 1 1 0 0 0 0 1 1 0 0 1 0 1 0 1 1 1 1 1 1 1 1 1 1 1 1]
just for better-visualization: invert clusters (permutation)
[1 1 0 1 1 1 1 1 0 0 1 1 1 1 0 0 1 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0]
0.771725032425
0.722546051351
```
That's a pretty much perfect fit to the ground-truth!


Question:
How to apply LabelEncoder for a specific column in Pandas dataframe I have a dataset loaded by dataframe where the class label needs to be encoded using LabelEncoder from scikit-learn. The column label is the class label column which has the following classes:
```
[‘Standing’, ‘Walking’, ‘Running’, ‘null’]
```
To perform label encoding, I tried the following but it does not work. How can I fix it?
```
from sklearn import preprocessing
import pandas as pd

df = pd.read_csv('dataset.csv', sep=',')
df.apply(preprocessing.LabelEncoder().fit_transform(df['label']))
```

Response:
You can try as following:
```
le = preprocessing.LabelEncoder()
df['label'] = le.fit_transform(df.label.values)
```
Or following would work too:
```
df['label'] = le.fit_transform(df['label'])
```
It will replace original label values in dataframe with encoded labels.


Question: Information Gain calculation with Scikit-learn I am using Scikit-learn for text classification. I want to calculate the Information Gain for each attribute with respect to a class in a (sparse) document-term matrix. the Information Gain is defined as H(Class) - H(Class | Attribute), where H is the entropy. in weka, this would be calculated with InfoGainAttribute. But I haven't found this measure in scikit-learn. (It was suggested that the formula above for Information Gain is the same measure as mutual information. This matches also the definition in wikipedia. Is it possible to use a specific setting for mutual information in scikit-learn to accomplish this task?)

Response:
You can use scikit-learn's mutual_info_classif here is an example
```
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_selection import mutual_info_classif
from sklearn.feature_extraction.text import CountVectorizer

categories = ['talk.religion.misc',
              'comp.graphics', 'sci.space']
newsgroups_train = fetch_20newsgroups(subset='train',
                                      categories=categories)

X, Y = newsgroups_train.data, newsgroups_train.target
cv = CountVectorizer(max_df=0.95, min_df=2,
                                     max_features=10000,
                                     stop_words='english')
X_vec = cv.fit_transform(X)

res = dict(zip(cv.get_feature_names(),
               mutual_info_classif(X_vec, Y, discrete_features=True)
               ))
print(res)
```
this will output a dictionary of each attribute, i.e. item in the vocabulary as keys and their information gain as values here is a sample of the output
```
{'bible': 0.072327479595571439,
 'christ': 0.057293733680219089,
 'christian': 0.12862867565281702,
 'christians': 0.068511328611810071,
 'file': 0.048056478042481157,
 'god': 0.12252523919766867,
 'gov': 0.053547274485785577,
 'graphics': 0.13044709565039875,
 'jesus': 0.09245436105573257,
 'launch': 0.059882179387444862,
 'moon': 0.064977781072557236,
 'morality': 0.050235104394123153,
 'nasa': 0.11146392824624819,
 'orbit': 0.087254803670582998,
 'people': 0.068118370234354936,
 'prb': 0.049176995204404481,
 'religion': 0.067695617096125316,
 'shuttle': 0.053440976618359261,
 'space': 0.20115901737978983,
 'thanks': 0.060202010019767334}
```


Question:
Distinguishing overfitting vs good prediction These are questions on how to calculate & reduce overfitting in machine learning. I think many new to machine learning will have the same questions, so I tried to be clear with my examples and questions in hope that answers here can help others. I have a very small sample of texts and I'm trying to predict values associated with them. I've used sklearn to calculate tf-idf, and insert those into a regression model for prediction. This gives me 26 samples with 6323 features - not a lot.. I know:
```
>> count_vectorizer = CountVectorizer(min_n=1, max_n=1)
>> term_freq = count_vectorizer.fit_transform(texts)
>> transformer = TfidfTransformer()
>> X = transformer.fit_transform(term_freq)
>> print X.shape

(26, 6323)
```
Inserting those 26 samples of 6323 features (X) and associated scores (y), into a LinearRegression model, gives good predictions. These are obtained using leave-one-out cross validation, from cross_validation.LeaveOneOut(X.shape[0], indices=True) :
```
using ngrams (n=1):
     human  machine  points-off  %error
      8.67    8.27    0.40       1.98
      8.00    7.33    0.67       3.34
      ...     ...     ...        ...
      5.00    6.61    1.61       8.06
      9.00    7.50    1.50       7.50
mean: 7.59    7.64    1.29       6.47
std : 1.94    0.56    1.38       6.91
```
Pretty good! Using ngrams (n=300) instead of unigrams (n=1), similar results occur, which is obviously not right. No 300-words occur in any of the texts, so the prediction should fail, but it doesn't:
```
using ngrams (n=300):
      human  machine  points-off  %error
       8.67    7.55    1.12       5.60
       8.00    7.57    0.43       2.13
       ...     ...     ...        ...
mean:  7.59    7.59    1.52       7.59
std :  1.94    0.08    1.32       6.61
```
Question 1: This might mean that the prediction model is overfitting the data. I only know this because I chose an extreme value for the ngrams (n=300) which I KNOW can't produce good results. But if I didn't have this knowledge, how would you normally tell that the model is over-fitting? In other words, if a reasonable measure (n=1) were used, how would you know that the good prediction was a result of being overfit vs. the model just working well? Question 2: What is the best way of preventing over-fitting (in this situation) to be sure that the prediction results are good or not? Question 3: If LeaveOneOut cross validation is used, how can the model possibly over-fit with good results? Over-fitting means the prediction accuracy will suffer - so why doesn't it suffer on the prediction for the text being left out? The only reason I can think of: in a tf-idf sparse matrix of mainly 0s, there is strong overlap between texts because so many terms are 0s - the regression then thinks the texts correlate highly. Please answer any of the questions even if you don't know them all. Thanks!

Response: how would you normally tell that the model is over-fitting? One useful rule of thumb is that you may be overfitting when your model's performance on its own training set is much better than on its held-out validation set or in a cross-validation setting. That's not all there is to it, though. The blog entry I linked to describes a procedure for testing for overfit: plot training set and validation set error as a function of training set size. If they show a stable gap at the right end of the plot, you're probably overfitting. What is the best way of preventing over-fitting (in this situation) to be sure that the prediction results are good or not? Use a held-out test set. Only do evaluation on this set when you're completely done with model selection (hyperparameter tuning); don't train on it, don't use it in (cross-)validation. The score you get on the test set is the model's final evaluation. This should show whether you've accidentally overfit the validation set(s). [Machine learning conferences are sometimes set up like a competition, where the test set is not given to the researchers until after they've delivered their final model to the organisers. In the meanwhile, they can use the training set as they please, e.g. by testing models using cross-validation. Kaggle does something similar.] If LeaveOneOut cross validation is used, how can the model possibly over-fit with good results? Because you can tune the model as much as you want in this cross-validation setting, until it performs nearly perfectly in CV. As an extreme example, suppose that you've implemented an estimator that is essentially a random number generator. You can keep trying random seeds until you hit a "model" that produces very low error in cross-validation, but that doesn't you've hit the right model. It means you've overfit to the cross-validation. See also this interesting warstory.


Question:
tf-idf feature weights using sklearn.feature_extraction.text.TfidfVectorizer this page: http://scikit-learn.org/stable/modules/feature_extraction.html mentions: As tf–idf is a very often used for text features, there is also another class called TfidfVectorizer that combines all the option of CountVectorizer and TfidfTransformer in a single model. then I followed the code and use fit_transform() on my corpus. How to get the weight of each feature computed by fit_transform()? I tried:
```

```python
vectorizer.idf_
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-39-5475eefe04c0> in <module>()
----> 1 vectorizer.idf_

AttributeError: 'TfidfVectorizer' object has no attribute 'idf_'
```
but this attribute is missing. Thanks
```

Response:
Since version 0.15, the tf-idf score of each feature can be retrieved via the attribute idf_ of the TfidfVectorizer object:
```
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = ["This is very strange",
          "This is very nice"]
vectorizer = TfidfVectorizer(min_df=1)
X = vectorizer.fit_transform(corpus)
idf = vectorizer.idf_
print dict(zip(vectorizer.get_feature_names(), idf))
```
Output:
```
{u'is': 1.0,
 u'nice': 1.4054651081081644,
 u'strange': 1.4054651081081644,
 u'this': 1.0,
 u'very': 1.0}
```
As discussed in the comments, prior to version 0.15, a workaround is to access the attribute idf_ via the supposedly hidden _tfidf (an instance of TfidfTransformer) of the vectorizer:
```
idf = vectorizer._tfidf.idf_
print dict(zip(vectorizer.get_feature_names(), idf))
```
which should give the same output as above.


Question: Scikit Learn GridSearchCV without cross validation (unsupervised learning) Is it possible to use GridSearchCV without cross validation? I am trying to optimize the number of clusters in KMeans clustering via grid search, and thus I don't need or want cross validation. The documentation is also confusing me because under the fit() method, it has an option for unsupervised learning (says to use None for unsupervised learning). But if you want to do unsupervised learning, you need to do it without cross validation and there appears to be no option to get rid of cross validation.

Response:
After much searching, I was able to find this thread. It appears that you can get rid of cross validation in GridSearchCV if you use: cv=[(slice(None), slice(None))] I have tested this against my own coded version of grid search without cross validation and I get the same results from both methods. I am posting this answer to my own question in case others have the same issue. Edit: to answer jjrr's question in the comments, here is an example use case:
```
from sklearn.metrics import silhouette_score as sc

def cv_silhouette_scorer(estimator, X):
    estimator.fit(X)
    cluster_labels = estimator.labels_
    num_labels = len(set(cluster_labels))
    num_samples = len(X.index)
    if num_labels == 1 or num_labels == num_samples:
        return -1
    else:
        return sc(X, cluster_labels)

cv = [(slice(None), slice(None))]
gs = GridSearchCV(estimator=sklearn.cluster.MeanShift(), param_grid=param_dict,
                  scoring=cv_silhouette_scorer, cv=cv, n_jobs=-1)
gs.fit(df[cols_of_interest])
```


Question:
PCA projection and reconstruction in scikit-learn I can perform PCA in scikit by code below: X_train has 279180 rows and 104 columns.
```
from sklearn.decomposition import PCA
pca = PCA(n_components=30)
X_train_pca = pca.fit_transform(X_train)
```
Now, when I want to project the eigenvectors onto feature space, I must do following:
```
""" Projection """
comp = pca.components_ #30x104
com_tr = np.transpose(pca.components_) #104x30
proj = np.dot(X_train,com_tr) #279180x104 * 104x30 = 297180x30
```
But I am hesitating with this step, because Scikit documentation says: components_: array, [n_components, n_features] Principal axes in feature space, representing the directions of maximum variance in the data. It seems to me, that it is already projected, but when I checked the source code, it returns only the eigenvectors. What is the right way how to project it? Ultimately, I am aiming to calculate the MSE of reconstruction.
```
""" Reconstruct """
recon = np.dot(proj,comp) #297180x30 * 30x104 = 279180x104

"""  MSE Error """
print "MSE = %.6G" %(np.mean((X_train - recon)**2))
```

Response:
You can do
```
proj = pca.inverse_transform(X_train_pca)
```
That way you do not have to worry about how to do the multiplications. What you obtain after pca.fit_transform or pca.transform are what is usually called the "loadings" for each sample, meaning how much of each component you need to describe it best using a linear combination of the components_ (the principal axes in feature space). The projection you are aiming at is back in the original signal space. This means that you need to go back into signal space using the components and the loadings. So there are three steps to disambiguate here. Here you have, step by step, what you can do using the PCA object and how it is actually calculated: pca.fit estimates the components (using an SVD on the centered Xtrain):
```
from sklearn.decomposition import PCA
 import numpy as np
 from numpy.testing import assert_array_almost_equal

 #Should this variable be X_train instead of Xtrain?
 X_train = np.random.randn(100, 50)

 pca = PCA(n_components=30)
 pca.fit(X_train)

 U, S, VT = np.linalg.svd(X_train - X_train.mean(0))

 assert_array_almost_equal(VT[:30], pca.components_)
```
pca.transform calculates the loadings as you describe
```
X_train_pca = pca.transform(X_train)

 X_train_pca2 = (X_train - pca.mean_).dot(pca.components_.T)

 assert_array_almost_equal(X_train_pca, X_train_pca2)
```
pca.inverse_transform obtains the projection onto components in signal space you are interested in
```
X_projected = pca.inverse_transform(X_train_pca)
 X_projected2 = X_train_pca.dot(pca.components_) + pca.mean_

 assert_array_almost_equal(X_projected, X_projected2)
```
You can now evaluate the projection loss
```
loss = np.sum((X_train - X_projected) ** 2, axis=1).mean()
```


Question: What is the meaning of the nu parameter in Scikit-Learn's SVM class? I am following the example shown in http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html#example-svm-plot-oneclass-py, where a one class SVM is used for anomaly detection. Now, this may be a notation unique to scikit-learn, but I couldn't find an explanation of how to use the parameter nu given to the OneClassSVM constructor. In http://scikit-learn.org/stable/modules/svm.html#nusvc, it is stated that the parameter nu is a reparametrization of the parameter C (which is the regularization parameter which I am familiar with) - but doesn't state how to perform that reparameterization. Both a formula and an intuition will be much appreciated. Thanks!

Response: The problem with C and the introduction of nu The problem with the parameter C is: that it can take any positive value that it has no direct interpretation. It is therefore hard to choose correctly and one has to resort to cross validation or direct experimentation to find a suitable value. In response Schölkopf et al. reformulated SVM to take a new regularization parameter nu. This parameter is: bounded between 0 and 1 has a direct interpretation Interpretation of nu The parameter nu is an upper bound on the fraction of margin errors and a lower bound of the fraction of support vectors relative to the total number of training examples. For example, if you set it to 0.05 you are guaranteed to find at most 5% of your training examples being misclassified (at the cost of a small margin, though) and at least 5% of your training examples being support vectors. Relationship between C and nu The relation between C and nu is governed by the following formula: nu = A+B/C A and B are constants which are unfortunately not that easy to calculate. Conclusion The takeaway message is that C and nu SVM are equivalent regarding their classification power. The regularization in terms of nu is easier to interpret compared to C, but the nu SVM is usually harder to optimize and runtime doesn't scale as well as the C variant with number of input samples. More details (including formulas for A and B) can be found here: Chang CC, Lin CJ - "Training nu-support vector classifiers: theory and algorithms"


Question:
ValueError: setting an array element with a sequence. while using SVM in scikit-learn I have been working on scikit-learn SVMs for a binary classification problem. I have calculated the features of audio files and wrote them into a CSV file. This is how each row in a CSV file looks like:
```
"13_10 The Long And Winding Road " "[-6.5633095666136669e-16,-1.56E-15,-3.21E-15,-2.20E-
15,-2.52E-15,-3.04E-15,-3.39E-15,-3.47E-15,-3.07E-15,-6.02E-15,-3.00E-15,-4.77E-15,-3.05E-
15,-2.13E-15,-1.57E-15,-1.87E-15,-2.05E-15,-1.76E-15,-1.38E-15,-9.89E-16,-7.89E-16,-8.99E-
16,-1.09E-15,-7.26E-16,-8.68E-16,-4.68E-16,-2.82E-16,-1.99E-16,-1.75E-16,-2.18E-16,-1.43E-
16,-1.56E-16,-1.91E-16,-1.21E-16,-4.82E-17,-4.39E-17,-2.89E-17,-2.05E-17,0.0]" 0
```
The first column has the name of the Audio, second column has the feature array and the last element is the label {0,1} for binary classification. There are 39 float values in the array. I am using the following code to extract them from the CSV file.
```
with open('File.csv', 'rb') as csvfile:
    albumreader = csv.reader(csvfile, delimiter=' ')
    data = list()
    for row in albumreader:
        data.append(row[0:])
data = np.array(data)

X_train = list()
Y_train = list()
k = data.shape[0]

for i in range(k):
    feature = data[i][1]
    x = map(float, feature[1:-2].split(','))
    X_train.append(x)
    label = data[i][2]
    y = float(label)
    Y_train.append(y)
```
So when I print X_train and Y_train I get exact values in an array. But when I use
```
clf = svm.SVC(C=1.0, cache_size=200,kernel='linear', max_iter=-1)
clf.fit(X_train,Y_train)
```
I get the error saying
```
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-
packages\spyderlib\widgets\externalshell\sitecustomize.py",line 540, in runfile
execfile(filename, namespace)
File "SVM_test.py", line 55, in <module>
clf.fit(X_train,Y_train)
File "sklearn\svm\base.py", line 137, in fit
File "sklearn\utils\validation.py", line 165, in atleast2d_or_csr
File "sklearn\utils\validation.py", line 142, in _atleast2d_or_sparse
File "sklearn\utils\validation.py", line 120, in array2d
File "C:\Python27\lib\site-packages\numpy\core\numeric.py", line 460, in asarray
return array(a, dtype, copy=False, order=order)
ValueError: setting an array element with a sequence.
```
Can someone help me as to what I can do now? I am really not sure what is happening inside. Both the dimensions of X_train and Y_train are same [X_train has 21 vectors with 39 elements and Y_train has 21 floats {0 or 1}, I don't see what made these errors. Note: I have a feeling that something might be wrong while I convert the numpy array to string and then a string to a numpy array. Thanks in advance. Edit: X_Train is very large. Here it is..
```
[[93812.4999999983, 73189.57452, 48892.17363, 37682.69053, 33709.51536, 20815.68443,   12476.88854, 13364.13645, 9574.010981, 5844.293383, 7910.017736, 12721.38592, 14184.99241, 6988.131481, 9407.380437, 6333.852471, 5688.156663, 7167.61338, 6911.084942, 9210.064235, 5732.338515, 3585.039683, 4433.278772, 4757.658741, 3387.832928, 2711.640327, 2680.255742, 1649.410788, 2024.333977, 997.2348795, 1102.115501, 1386.86396, 1160.477719, 883.941971, 881.2712624, 749.3620066, 885.6355941, 514.1635441, 0.0], [93411.33935126709, 90714.51224, 89773.71828, 61018.71033, 28082.94493, 10120.93228, 11106.07725, 6204.140734, 5968.528906, 4970.099848, 6967.870007, 6990.611982, 7656.630743, 6615.957476, 5573.621516, 8957.245225, 8512.408652, 6976.021692, 7774.215884, 5301.046573, 4666.784091, 2539.587812, 2953.578612, 3529.863917, 2365.101263, 2579.870258, 2890.325096, 3302.179572, 2078.005268, 1425.18236, 1297.961119, 736.4896705, 640.0635888, 819.022382, 659.9559469, 438.2773842, 359.3957991, 193.9937669, 0.0], [95528.45827960827, 79000.64725, 75540.32258, 47915.39365, 29573.63325, 13554.15721, 10101.04124, 6935.685456, 13681.96711, 7726.754596, 9413.96529, 9468.785586, 10479.23762, 10070.81121, 8893.475453, 9517.553541, 8493.077533, 8021.721412, 8568.069341, 7687.282084, 9902.16325, 5442.263263, 5575.258138, 4748.557573, 4580.647869, 3014.91771, 3958.708771, 2851.846841, 3407.31788, 1982.369432, 1937.459179, 1689.049684, 1457.579778, 1055.411047, 1048.471861, 661.6174333, 827.8371903, 414.802354], [101683.46698748806, 62367.04137, 66444.15995, 49621.45404, 31623.19485, 16585.34427, 12271.46378, 12114.5615, 6666.281052, 9335.886213, 19314.70299, 22588.00911, 14133.31813, 12723.03772, 7994.399321, 11447.449, 15457.39519, 7419.208867, 9286.751692, 6128.746537, 5617.886066, 4461.131891, 4651.73188, 5835.270092, 3876.10397, 4499.228748, 2661.999151, 1431.362029, 1378.115091, 1048.827946, 1470.297845, 1087.453644, 825.6318213, 861.5003481, 804.8519616, 397.0719915, 368.8037827, 293.36727], [96614.66763477474, 89674.79785, 73045.22026, 55387.48162, 32450.76131, 26161.93729, 16379.95699, 13446.77762, 6178.297767, 4499.9064, 6128.624979, 4928.968691, 7139.579976, 6442.404748, 7303.917218, 9064.476552, 8246.412739, 4526.169172, 4931.980606, 4022.38625, 3193.080061, 3991.709836, 4894.262891, 4523.545798, 5013.65655, 3165.268896, 2252.272798, 1971.857637, 1543.455559, 1248.305408, 1340.303682, 1069.466847, 1062.971087, 596.4763587, 541.7390803, 481.9598053, 261.6165905, 135.050925], [77116.86410716272, 85174.88022, 48949.81474, 39272.16867, 28721.41507, 26604.82082, 17057.75385, 11417.45143, 12775.94149, 8095.318819, 8318.738856, 7768.406613, 9501.155323, 8215.579012, 5801.439936, 6997.611748, 8358.126592, 6710.072432, 7903.976639, 4770.389995, 4443.449546, 3622.278619, 3628.985312, 4025.879147, 3378.124716, 1681.144815, 1873.675902, 1813.454359, 1203.261884, 734.9896092, 612.7767898, 581.1641439, 554.9952946, 338.9208239, 329.6306536, 210.3361409, 124.684456, 95.1698974], [86000.24134707314, 54315.80346, 61723.06357, 48194.93238, 34145.18298, 18060.21908, 17759.95552, 13594.71484, 10034.81255, 6892.428679, 13609.12234, 11345.97425, 12640.27575, 13636.73634, 8353.154837, 11543.51778, 9620.892875, 5364.536625, 6645.647746, 6939.929388, 6404.367983, 4279.002491, 5473.449778, 5173.72645, 4161.012572, 3189.349797, 1868.016199, 2370.813774, 1991.805589, 1862.750613, 1535.097522, 1195.019326, 824.4997101, 836.5762868, 758.8865079, 739.0096703, 426.339462, 495.362511], [88356.8775920093, 68677.18631, 56499.17126, 41069.83582, 34004.99481, 21584.94408, 16827.63584, 10875.88263, 8838.404327, 10399.33201, 10247.97332, 11592.57345, 6888.99984, 8027.86374, 4396.353004, 4926.542018, 4160.408132, 4829.051031, 5104.507749, 4445.908694, 4113.401198, 2070.059053, 2331.063956, 3091.764189, 2708.490628, 1357.792132, 1476.379979, 1099.46743, 895.2046416, 1017.410994, 855.9326154, 807.2299975, 817.8896259, 688.1633806, 620.1147918, 404.4791452, 355.3012015, 155.124636], [129161.3158422606, 99871.12426, 69682.53863, 42152.57846, 27722.10719, 16851.46834, 12503.65957, 15820.8482, 10208.86252, 3737.281589, 11388.29292, 9216.418551, 8412.969115, 8915.691889, 7214.795344, 6312.935476, 5691.760401, 4452.333587, 6080.803383, 3169.211512, 4640.513939, 2965.070935, 2603.678979, 3427.596811, 2650.097593, 3407.197764, 2399.210804, 1585.540133, 900.6057596, 1562.799097, 1414.458688, 1085.727804, 862.853398, 1046.809149, 1299.422095, 452.1395434, 416.0278005, 342.487369], [97676.58730158686, 85928.37013, 60031.54702, 50283.65633, 30440.49477, 23396.44028, 17693.84492, 13834.72723, 13079.6, 9484.172923, 11026.12866, 15489.77935, 14751.23748, 7719.575611, 6916.062149, 9947.922301, 9860.230801, 6685.554777, 5314.504743, 6412.026375, 5126.472976, 3994.412881, 3469.94381, 3087.75188, 2150.012155, 2510.441776, 1633.896465, 1468.22101, 1451.997957, 1594.288508, 1208.749937, 1539.411357, 846.1440547, 1015.738147, 760.9050287, 531.4752058, 352.2906744, 256.992846], [99873.48353552721, 96128.33417, 56062.95108, 48316.51261, 33803.61475, 20090.40769, 14532.69355, 16973.62408, 11745.412, 10555.56359, 12415.12332, 11311.00716, 13055.02538, 13457.43473, 11949.02017, 13726.34027, 13210.19444, 6924.913491, 7526.293551, 6489.797287, 7504.193589, 3693.345327, 3173.144967, 4589.951959, 3817.607517, 2296.577132, 4241.66248, 2298.259695, 2104.233705, 1894.800787, 1435.902299, 1237.861542, 1008.052264, 743.557111, 447.3644689, 360.231905, 263.6887002, 252.53243], [118318.40927047582, 96894.04475, 72455.95855, 53538.90521, 34270.2485, 14028.66282, 6110.994324, 10831.06944, 6500.061124, 5648.546259, 9746.722376, 11098.67455, 12414.31738, 11859.15818, 5661.36057, 6467.490449, 7160.019668, 4986.101354, 4805.715894, 4384.860917, 4818.433908, 2776.480858, 2906.711958, 4180.355966, 3029.563639, 2121.677425, 2977.055372, 1650.875378, 1328.284924, 1641.967101, 1374.844716, 1269.983055, 756.2822371, 746.9782069, 635.1025738, 901.5181204, 500.4240422, 124.234986], [99496.1074660524, 91134.19642, 64615.65163, 51749.95315, 27017.75136, 17498.19736, 8686.464718, 6354.494714, 6279.181765, 6011.661362, 9583.683802, 16802.58819, 12848.82539, 12448.85086, 9717.906293, 6025.712047, 8968.944145, 6116.427844, 8009.500521, 5857.252734, 5994.629798, 4602.865888, 5568.279578, 3847.961198, 3664.838032, 2285.641295, 2343.300802, 1538.656643, 1595.004126, 1438.685894, 1278.233128, 1138.847548, 1387.660031, 727.3346259, 443.3437923, 399.422316, 202.3671643, 210.818774], [97897.81181619188, 81534.24658, 86124.34023, 55859.41234, 43498.35095, 16317.93548, 9240.704588, 8335.639737, 5398.77203, 2959.587234, 7638.934756, 9237.569061, 9669.92492, 6395.762472, 5297.481894, 4628.757031, 5965.00084, 5360.168945, 4918.802753, 5403.035015, 7760.124783, 4316.46269, 3586.003412, 4862.517393, 2722.334238, 1950.153709, 2308.64693, 1738.602095, 1431.956923, 1195.875585, 903.5619486, 628.8441079, 378.5951575, 279.5559759, 290.8523867, 185.8872588, 124.4224622, 102.6474251], [92929.03125177219, 66037.16827, 85713.22692, 60594.81708, 21299.03928, 9728.745394, 7164.560274, 7530.287996, 3986.197072, 4768.423334, 7965.588661, 6884.742393, 7813.113615, 6783.772795, 5068.375149, 5563.205324, 4549.089711, 4178.977925, 7176.864923, 3595.204266, 4075.654498, 3667.874878, 5018.867408, 4632.204595, 4236.022945, 2419.634542, 1965.732854, 2017.314496, 1125.444672, 1776.994722, 1380.972752, 877.5693874, 1048.039171, 698.4293241, 587.3589805, 425.3561446, 374.9688448, 242.143167], [112279.4547224921, 85626.58906, 82479.88981, 41194.72139, 18581.67331, 17171.661, 11041.06798, 7470.697485, 5647.489476, 5413.921458, 6258.45235, 7817.02576, 5690.588758, 5018.057148, 2835.675844, 4192.365122, 5264.669752, 2899.863762, 4722.075443, 4359.368543, 4475.52712, 4364.193393, 4366.760559, 4466.265791, 3581.127965, 3229.694902, 3061.592084, 2761.368431, 2924.520852, 2278.74424, 1842.130366, 1353.160812, 1061.970453, 801.4987863, 559.7692834, 542.6125554, 365.0923416, 345.207555], [103517.74691357967, 75660.08945, 77823.62831, 44178.34395, 30627.84085, 16822.14795, 12153.82383, 10477.03604, 6737.154621, 3948.567091, 5952.492101, 6657.190597, 8458.524435, 4644.542091, 3262.595869, 6196.748153, 4725.493005, 3131.648336, 3043.832975, 2397.211069, 2221.444205, 1846.007568, 1906.256992, 2565.899774, 1879.678929, 1983.431392, 2057.925713, 1379.158985, 1161.566123, 1269.932159, 1882.60896, 2175.463202, 1945.131584, 2617.451168, 1724.479089, 934.2682688, 703.1608361, 325.546], [93568.70860926574, 94747.49539969431, 46432.77848910925, 34021.920729891295, 20420.991633759644, 11780.466421174959, 11808.677934216039, 8356.053623407755, 5251.866299007888, 1837.1346095714694, 3388.9444867864604, 4160.840941722876, 3099.1858407062873, 2498.7020047692304, 2320.2543950190047, 3123.103649596546, 2353.3994748227874, 2282.188646923959, 2461.343070326571, 1867.3943363024234, 2097.623570329987, 2009.6550578189285, 3308.2027220730074, 2765.1388333698114, 1557.1149504889588, 1365.611056602633, 1960.1916988919656, 1357.4554303292293, 1174.7058492639005, 1132.4243713198962, 845.4972050001261, 1168.2703928255426, 792.9426082625839, 670.3864757268884, 462.94718979251763, 418.287362938786, 440.0999154920886, 177.49705973335398, 0.0], [95355.29766162521, 80982.187596427, 52611.83577098383, 32094.021907352668, 17954.16900608238, 10539.432714398477, 8455.780912660928, 7826.206728864228, 6509.019127983875, 3428.593131775805, 4133.834750579424, 4897.866949416399, 4527.962919676826, 3097.9755890532115, 2644.294656259542, 3715.9623636641186, 2820.3307205895694, 2502.4555417041665, 4294.009887075389, 3305.480815069842, 3473.739729060158, 3436.008663252062, 2646.057627969427, 2915.118003316749, 2807.214040627724, 2182.2047542975124, 2307.7279832228096, 2051.914227220658, 1701.1785697138466, 1387.86622139378, 1717.4780638249865, 1444.4320186566786, 1543.3397450160378, 1008.7972827019012, 804.7763630817929, 727.0076251244793, 661.7971983605773, 328.023137248546, 0.0], [106950.75545607107, 83171.33894927795, 98570.84168082179, 53995.601284217235, 34045.2113137451, 32682.002511908893, 20258.01771044016, 17863.78159524713, 10999.026649078776, 7606.650910143417, 8186.182643934389, 12307.240199947704, 6014.871257290792, 4781.08981401508, 5131.609324634855, 4391.107045269739, 4364.496837469433, 3795.810404058682, 5693.929878923241, 3511.0866864164072, 4967.40355405853, 3290.291028496737, 2401.232195128987, 2787.2578565673602, 2210.985797970096, 2106.714353398232, 1799.725035771931, 2223.1076215378416, 1189.234114777526, 1003.1624544891614, 1046.7700894681655, 812.1805254193989, 750.3209854314467, 893.172975198784, 492.44092578555313, 379.87738447537436, 169.4616484512177, 100.56120686339501, 0.0]]
```
Y_Train is just the labels for individual sets of features. It is like this:
```
[0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0]
```
Hope that helps!

Response: Finally I found the answer to my question with the help of some ideas from @larsmans and @eickenberg. The problem was that X_train did not have the same number of elements in all the arrays. So, it was not able to form a 2D array. Now that I have added an additional value to that array, the dimensionality matched for all the 1D arrays and X_train was able to form a 2D array. Thanks for the ideas guys!


Question:
KeyError when loading pickled scikit-learn model using joblib I have an object that contains within it two scikit-learn models, an IsolationForest and a RandomForestClassifier, that I would like to pickle and later unpickle and use to produce predictions. Apart from the two models, the object contains a couple of StandardScalers and a couple of Python lists. Pickling this object using joblib is unproblematic, but when I try to unpickle it later I get the following exception:
```
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/home/(...)/python3.5/site-packages/joblib/numpy_pickle.py", line 578, in load
   obj = _unpickle(fobj, filename, mmap_mode)
 File "/home/(...)/python3.5/site-packages/joblib/numpy_pickle.py", line 508, in _unpickle
   obj = unpickler.load()
 File "/usr/lib/python3.5/pickle.py", line 1039, in load
   dispatch[key[0]](self)
KeyError: 0
```
The same application both pickles and unpickles the object, so the versions of scikit-learn, joblib and other libraries are the same. I'm not sure where to start debugging, given the vague error. Any ideas or pointers?

Response: The solution to this was pretty banal: Without being aware of it I was using the version of joblib in sklearn.externals.joblib for the pickling, but a newer version of joblib for unpickling the object. The problem was resolved when I used the newer version of joblib for both tasks.


Question:
TypeError: only integer arrays with one element can be converted to an index I'm getting the following error when performing recursive feature selection with cross-validation:
```
Traceback (most recent call last):
  File "/Users/.../srl/main.py", line 32, in <module>
    argident_sys.train_classifier()
  File "/Users/.../srl/identification.py", line 194, in train_classifier
    feat_selector.fit(train_argcands_feats,train_argcands_target)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn/feature_selection/rfe.py", line 298, in fit
    ranking_ = rfe.fit(X[train], y[train]).ranking_
TypeError: only integer arrays with one element can be converted to an index
```
The code that generates the error is the following:
```
def train_classifier(self):

    # Get the argument candidates
    argcands = self.get_argcands(self.reader)

    # Extract the necessary features from the argument candidates
    train_argcands_feats = []
    train_argcands_target = []

    for argcand in argcands:
        train_argcands_feats.append(self.extract_features(argcand))
        if argcand["info"]["label"] == "NULL":
            train_argcands_target.append("NULL")
        else:
            train_argcands_target.append("ARG")

    # Transform the features to the format required by the classifier
    self.feat_vectorizer = DictVectorizer()
    train_argcands_feats = self.feat_vectorizer.fit_transform(train_argcands_feats)

    # Transform the target labels to the format required by the classifier
    self.target_names = list(set(train_argcands_target))
    train_argcands_target = [self.target_names.index(target) for target in train_argcands_target]

    ## Train the appropriate supervised model

    # Recursive Feature Elimination
    self.classifier = LogisticRegression()
    feat_selector = RFECV(estimator=self.classifier, step=1, cv=StratifiedKFold(train_argcands_target, 10))

    feat_selector.fit(train_argcands_feats,train_argcands_target)

    print feat_selector.n_features_
    print feat_selector.support_
    print feat_selector.ranking_
    print feat_selector.cv_scores_

    return
```
I know I should also perform GridSearch for the parameters of the LogisticRegression classifier, but I don't think that's the source of the error (or is it?). I should mention that I'm testing with around 50 features, and almost all of them are categoric (that's why I use the DictVectorizer to transform them appropriately). Any help or guidance you could give me is more than welcome. Thanks! EDIT Here's some training data examples:
```
train_argcands_feats = [{'head_lemma': u'Bras\xedlia', 'head': u'Bras\xedlia', 'head_postag': u'PROP'}, {'head_lemma': u'Pesquisa_Datafolha', 'head': u'Pesquisa_Datafolha', 'head_postag': u'N'}, {'head_lemma': u'dado', 'head': u'dado', 'head_postag': u'N'}, {'head_lemma': u'postura', 'head': u'postura', 'head_postag': u'N'}, {'head_lemma': u'maioria', 'head': u'maioria', 'head_postag': u'N'}, {'head_lemma': u'querer', 'head': u'quer', 'head_postag': u'V-FIN'}, {'head_lemma': u'PT', 'head': u'PT', 'head_postag': u'PROP'}, {'head_lemma': u'participar', 'head': u'participando', 'head_postag': u'V-GER'}, {'head_lemma': u'surpreendente', 'head': u'supreendente', 'head_postag': u'ADJ'}, {'head_lemma': u'Bras\xedlia', 'head': u'Bras\xedlia', 'head_postag': u'PROP'}, {'head_lemma': u'Pesquisa_Datafolha', 'head': u'Pesquisa_Datafolha', 'head_postag': u'N'}, {'head_lemma': u'revelar', 'head': u'revela', 'head_postag': u'V-FIN'}, {'head_lemma': u'recusar', 'head': u'recusando', 'head_postag': u'V-GER'}, {'head_lemma': u'maioria', 'head': u'maioria', 'head_postag': u'N'}, {'head_lemma': u'PT', 'head': u'PT', 'head_postag': u'PROP'}, {'head_lemma': u'participar', 'head': u'participando', 'head_postag': u'V-GER'}, {'head_lemma': u'surpreendente', 'head': u'supreendente', 'head_postag': u'ADJ'}, {'head_lemma': u'Bras\xedlia', 'head': u'Bras\xedlia', 'head_postag': u'PROP'}, {'head_lemma': u'Pesquisa_Datafolha', 'head': u'Pesquisa_Datafolha', 'head_postag': u'N'}, {'head_lemma': u'revelar', 'head': u'revela', 'head_postag': u'V-FIN'}, {'head_lemma': u'governo', 'head': u'Governo', 'head_postag': u'N'}, {'head_lemma': u'de', 'head': u'de', 'head_postag': u'PRP'}, {'head_lemma': u'governo', 'head': u'Governo', 'head_postag': u'N'}, {'head_lemma': u'recusar', 'head': u'recusando', 'head_postag': u'V-GER'}, {'head_lemma': u'maioria', 'head': u'maioria', 'head_postag': u'N'}, {'head_lemma': u'querer', 'head': u'quer', 'head_postag': u'V-FIN'}, {'head_lemma': u'PT', 'head': u'PT', 'head_postag': u'PROP'}, {'head_lemma': u'surpreendente', 'head': u'supreendente', 'head_postag': u'ADJ'}, {'head_lemma': u'Bras\xedlia', 'head': u'Bras\xedlia', 'head_postag': u'PROP'}, {'head_lemma': u'Pesquisa_Datafolha', 'head': u'Pesquisa_Datafolha', 'head_postag': u'N'}, {'head_lemma': u'revelar', 'head': u'revela', 'head_postag': u'V-FIN'}, {'head_lemma': u'muito', 'head': u'Muitas', 'head_postag': u'PRON-DET'}, {'head_lemma': u'prioridade', 'head': u'prioridades', 'head_postag': u'N'}, {'head_lemma': u'com', 'head': u'com', 'head_postag': u'PRP'}, {'head_lemma': u'prioridade', 'head': u'prioridades', 'head_postag': u'N'}]

train_argcands_target = ['NULL', 'ARG', 'ARG', 'ARG', 'NULL', 'NULL', 'NULL', 'NULL', 'NULL', 'NULL', 'NULL', 'NULL', 'ARG', 'ARG', 'ARG', 'ARG', 'NULL', 'NULL', 'NULL', 'NULL', 'ARG', 'NULL', 'NULL', 'NULL', 'NULL', 'NULL', 'ARG', 'NULL', 'NULL', 'NULL', 'NULL', 'ARG', 'ARG', 'NULL', 'NULL']
```

Response: I finally got to solve the problem. Two things had to be done: train_argcands_target is a list and it has to be a numpy array. I'm surprised it worked well before when I just used the estimator directly. For some reason (I don't know why, yet), it doesn't work either if I use the sparse matrix created by the DictVectorizer. I had to, "manually", transform each feature dictionary to a feature array with just integers representing each feature value. The transformation process is similar to the one I present in the code for the target values. Thanks to everyone who tried to help!


Question:
Why does sklearn Imputer need to fit? I'm really new in this whole machine learning thing and I'm taking an online course on this subject. In this course, the instructors showed the following piece of code:
```
imputer = Inputer(missing_values = 'Nan', strategy = 'mean', axis=0)
imputer = Imputer.fit(X[:, 1:3])
X[:, 1:3] = imputer.transform(X[:, 1:3])
```
I don't really get why this imputer object needs to fit. I mean, I´m just trying to get rid of missing values in my columns by replacing them with the column mean. From the little I know about programming, this is a pretty simple, iterative procedure, and wouldn´t require a model that has to train on data to be accomplished. Can someone please explain how this imputer thing works and why it requires training to replace some missing values by the column mean? I have read sci-kit's documentation, but it just shows how to use the methods, and not why they´re required. Thank you.

Response:
The Imputer fills missing values with some statistics (e.g. mean, median, ...) of the data. To avoid data leakage during cross-validation, it computes the statistic on the train data during the fit, stores it and uses it on the test data, during the transform.
```py
from sklearn.preprocessing import Imputer
obj = Imputer(strategy='mean')

obj.fit([[1, 2, 3], [2, 3, 4]])
print(obj.statistics_)
# array([ 1.5,  2.5,  3.5])

X = obj.transform([[4, np.nan, 6], [5, 6, np.nan]])
print(X)
# array([[ 4. ,  2.5,  6. ],
#        [ 5. ,  6. ,  3.5]])
```
You can do both steps in one if your train and test data are identical, using fit_transform.
```py
X = obj.fit_transform([[1, 2, np.nan], [2, 3, 4]])
print(X)
# array([[ 1. ,  2. ,  4. ],
#        [ 2. ,  3. ,  4. ]])
```
This data leakage issue is important, since the data distribution may change from the training data to the testing data, and you don't want the information of the testing data to be already present during the fit. See the doc for more information about cross-validation.


Question: Data Standardization vs Normalization vs Robust Scaler I am working on data preprocessing and want to compare the benefits of Data Standardization vs Normalization vs Robust Scaler practically. In theory, the guidelines are: Advantages: Standardization: scales features such that the distribution is centered around 0, with a standard deviation of 1. Normalization: shrinks the range such that the range is now between 0 and 1 (or -1 to 1 if there are negative values). Robust Scaler: similar to normalization but it instead uses the interquartile range, so that it is robust to outliers. Disadvantages: Standardization: not good if the data is not normally distributed (i.e. no Gaussian Distribution). Normalization: get influenced heavily by outliers (i.e. extreme values). Robust Scaler: doesn't take the median into account and only focuses on the parts where the bulk data is. I created 20 random numerical inputs and tried the above-mentioned methods (numbers in red color represent the outliers): I noticed that -indeed- the Normalization got affected negatively by the outliers and the change scale between the new values became tiny (all values almost identical -6 digits after the decimal point- 0.000000x) even there is noticeable differences between the original inputs! My questions are: Am I right to say that also Standardization gets affected negatively by the extreme values as well? If not, why according to the result provided? I really can't see how the Robust Scaler improved the data because I still have extreme values in the resulted data set? Any simple complete interpretation?

Response: Am I right to say that also Standardization gets affected negatively by the extreme values as well? Indeed you are; the scikit-learn docs themselves clearly warn for such a case: However, when data contains outliers, StandardScaler can often be mislead. In such cases, it is better to use a scaler that is robust against outliers. More or less, the same holds true for the MinMaxScaler as well. I really can't see how the Robust Scaler improved the data because I still have extreme values in the resulted data set? Any simple -complete interpretation? Robust does not mean immune, or invulnerable, and the purpose of scaling is not to "remove" outliers and extreme values - this is a separate task with its own methodologies; this is again clearly mentioned in the relevant scikit-learn docs: RobustScaler [...] Note that the outliers themselves are still present in the transformed data. If a separate outlier clipping is desirable, a non-linear transformation is required (see below). where the "see below" refers to the QuantileTransformer and quantile_transform.


Question:
Multiclass classification with xgboost classifier? I am trying out multi-class classification with xgboost and I've built it using this code,
```
clf = xgb.XGBClassifier(max_depth=7, n_estimators=1000)

clf.fit(byte_train, y_train)
train1 = clf.predict_proba(train_data)
test1 = clf.predict_proba(test_data)
```
This gave me some good results. I've got log-loss below 0.7 for my case. But after looking through few pages I've found that we have to use another objective in XGBClassifier for multi-class problem. Here's what is recommended from those pages.
```
clf = xgb.XGBClassifier(max_depth=5, objective='multi:softprob', n_estimators=1000,
                        num_classes=9)

clf.fit(byte_train, y_train)
train1 = clf.predict_proba(train_data)
test1 = clf.predict_proba(test_data)
```
This code is also working but it's taking a lot of time to complete compared when to my first code. Why is my first code also working for multi-class case? I have checked that it's default objective is binary:logistic used for binary classification but it worked really well for multi-class? Which one should I use if both are correct?

Response:
In fact, even if the default obj parameter of XGBClassifier is binary:logistic, it will internally judge the number of class of label y. When the class number is greater than 2, it will modify the obj parameter to multi:softmax. https://github.com/dmlc/xgboost/blob/master/python-package/xgboost/sklearn.py
```py
class XGBClassifier(XGBModel, XGBClassifierBase):
    # pylint: disable=missing-docstring,invalid-name,too-many-instance-attributes
    def __init__(self, objective="binary:logistic", **kwargs):
        super().__init__(objective=objective, **kwargs)

    def fit(self, X, y, sample_weight=None, base_margin=None,
            eval_set=None, eval_metric=None,
            early_stopping_rounds=None, verbose=True, xgb_model=None,
            sample_weight_eval_set=None, callbacks=None):
        # pylint: disable = attribute-defined-outside-init,arguments-differ

        evals_result = {}
        self.classes_ = np.unique(y)
        self.n_classes_ = len(self.classes_)

        xgb_options = self.get_xgb_params()

        if callable(self.objective):
            obj = _objective_decorator(self.objective)
            # Use default value. Is it really not used ?
            xgb_options["objective"] = "binary:logistic"
        else:
            obj = None

        if self.n_classes_ > 2:
            # Switch to using a multiclass objective in the underlying
            # XGB instance
            xgb_options['objective'] = 'multi:softprob'
            xgb_options['num_class'] = self.n_classes_
```


Question: How to put more weight on certain features in machine learning? If using a library like scikit-learn, how do I assign more weight on certain features in the input to a classifier like SVM? Is this something people do or not?

Response: First of all - you should probably not do it. The whole concept of machine learning is to use statistical analysis to assign optimal weights. You are interfering here with the whole concept, thus you need really strong evidence that this is crucial to the process you are trying to model, and for some reason your model is currently missing it. That being said - there is no general answer. This is purely model specific, some of which will allow you to weight features - in random forest you could bias distribution from which you sample features to analyse towards the ones that you are interested in; in SVM it should be enough to just multiply given feature by a constant - remember when you were told to normalize your features in SVM? This is why - you can use the scale of features to 'steer' your classifier towards given features. The ones with high values will be preffered. This will actually work for most linear weight norm-regularized models (regularized logistic regression, ridge regression, lasso etc.).


Question: What is the difference between sample weight and class weight options in scikit learn? I have class imbalance problem and want to solve this using cost sensitive learning. under sample and over sample give weights to class to use a modified loss function Question Scikit learn has 2 options called class weights and sample weights. Is sample weight actually doing option 2) and class weight options 1). Is option 2) the the recommended way of handling class imbalance.

Response: It's similar concepts, but with sample_weights you can force estimator to pay more attention on some samples, and with class_weights you can force estimator to learn with attention to some particular class. sample_weight=0 or class_weight=0 basically means that estimator doesn't need to take into consideration such samples/classes in learning process at all. Thus classifier (for example) will never predict some class if class_weight = 0 for this class. If some sample_weight/class_weight bigger than sample_weight/class_weight on other samples/classes - estimator will try to minimize error on that samples/classes in the first place. You can use user-defined sample_weights and class_weights simultaneously. If you want to undersample/oversample your training set with simple cloning/removing - this will be equal to increasing/decreasing of corresponding sample_weights/class_weights. In more complex cases you can also try artificially generate samples, with techniques like SMOTE.


Question: Using cosine distance with scikit learn KNeighborsClassifier Is it possible to use something like 1 - cosine similarity with scikit learn's KNeighborsClassifier? This answer says no, but on the documentation for KNeighborsClassifier, it says the metrics mentioned in DistanceMetrics are available. Distance metrics don't include an explicit cosine distance, probably because it's not really a distance, but supposedly it's possible to input a function into the metric. I tried inputting the scikit learn linear kernel into KNeighborsClassifier but it gives me an error that the function needs two arrays as arguments. Anyone else tried this?

Response: The cosine similarity is generally defined as xT y / (||x|| * ||y||), and outputs 1 if they are the same and goes to -1 if they are completely different. This definition is not technically a metric, and so you can't use accelerating structures like ball and kd trees with it. If you force scikit learn to use the brute force approach, you should be able to use it as a distance if you pass it your own custom distance metric object. There are methods of transforming the cosine similarity into a valid distance metric if you would like to use ball trees (you can find one in the JSAT library) Notice though, that xT y / (||x|| * ||y||) = (x/||x||)T (y/||y||). The euclidean distance can be equivalently written as sqrt(xTx + yTy − 2 xTy). If we normalize every datapoint before giving it to the KNeighborsClassifier, then x^T x = 1 for all x. So the euclidean distance will degrade to sqrt(2 − 2x^T y). For completely the same inputs, we would get sqrt(2-2*1) = 0 and for complete opposites sqrt(2-2*-1)= 2. And it is clearly a simple shape, so you can get the same ordering as the cosine distance by normalizing your data and then using the euclidean distance. So long as you use the uniform weights option, the results will be identical to having used a correct Cosine Distance.


Question:
sklearn: Turning off warnings When I'm fitting sklearn's LogisticRegression using a 1 column python pandas DataFrame (not a Series object), I get this warning:
```
/Library/Python/2.7/site-packages/sklearn/preprocessing/label.py:125:
DataConversionWarning: A column-vector y was passed when a 1d array was
expected. Please change the shape of y to (n_samples, ), for example using
ravel().
y = column_or_1d(y, warn=True)
```
I know I could easily advert this warning in my code, but how can I turn off these warnings?

Response:
You can use this:
```
import warnings
from sklearn.exceptions import DataConversionWarning
warnings.filterwarnings(action='ignore', category=DataConversionWarning)
```