User guide: contents

• 1. Supervised learning
  • 1.1. Generalized Linear Models
    • 1.1.1. Ordinary Least Squares
      • 1.1.1.1. Ordinary Least Squares Complexity
    • 1.1.2. Ridge Regression
      • 1.1.2.1. Ridge Complexity
      • 1.1.2.2. Setting the regularization parameter: generalized Cross-Validation
    • 1.1.3. Lasso
      • 1.1.3.1. Setting regularization parameter
        • 1.1.3.1.1. Using cross-validation
        • 1.1.3.1.2. Information-criteria based model selection
    • 1.1.4. Elastic Net
    • 1.1.5. Multi-task Lasso
    • 1.1.6. Least Angle Regression
    • 1.1.7. LARS Lasso
      • 1.1.7.1. Mathematical formulation
    • 1.1.8. Orthogonal Matching Pursuit (OMP)
    • 1.1.9. Bayesian Regression
      • 1.1.9.1. Bayesian Ridge Regression
      • 1.1.9.2. Automatic Relevance Determination - ARD
    • 1.1.10. Logistic regression
    • 1.1.11. Stochastic Gradient Descent - SGD
    • 1.1.12. Perceptron
    • 1.1.13. Passive Aggressive Algorithms
  • 1.2. Support Vector Machines
    • 1.2.1. Classification
      • 1.2.1.1. Multi-class classification
      • 1.2.1.2. Scores and probabilities
      • 1.2.1.3. Unbalanced problems
    • 1.2.2. Regression
    • 1.2.3. Density estimation, novelty detection
    • 1.2.4. Complexity
    • 1.2.5. Tips on Practical Use
    • 1.2.6. Kernel functions
      • 1.2.6.1. Custom Kernels
        • 1.2.6.1.1. Using Python functions as kernels
        • 1.2.6.1.2. Using the Gram matrix
        • 1.2.6.1.3. Parameters of the RBF Kernel
    • 1.2.7. Mathematical formulation
      • 1.2.7.1. SVC
      • 1.2.7.2. NuSVC
    • 1.2.8. Implementation details
  • 1.3. Stochastic Gradient Descent
    • 1.3.1. Classification
    • 1.3.2. Regression
    • 1.3.3. Stochastic Gradient Descent for sparse data
    • 1.3.4. Complexity
    • 1.3.5. Tips on Practical Use
    • 1.3.6. Mathematical formulation
      • 1.3.6.1. SGD
    • 1.3.7. Implementation details
  • 1.4. Nearest Neighbors
    • 1.4.1. Unsupervised Nearest Neighbors
      • 1.4.1.1. Finding the Nearest Neighbors
      • 1.4.1.2. KDTree and BallTree Classes
    • 1.4.2. Nearest Neighbors Classification
    • 1.4.3. Nearest Neighbors Regression
    • 1.4.4. Nearest Neighbor Algorithms
      • 1.4.4.1. Brute Force
      • 1.4.4.2. K-D Tree
      • 1.4.4.3. Ball Tree
      • 1.4.4.4. Choice of Nearest Neighbors Algorithm
      • 1.4.4.5. Effect of leaf_size
    • 1.4.5. Nearest Centroid Classifier
      • 1.4.5.1. Nearest Shrunken Centroid
  • 1.5. Gaussian Processes
    • 1.5.1. Examples
      • 1.5.1.1. An introductory regression example
      • 1.5.1.2. Fitting Noisy Data
    • 1.5.2. Mathematical formulation
      • 1.5.2.1. The initial assumption
      • 1.5.2.2. The best linear unbiased prediction (BLUP)
      • 1.5.2.3. The empirical best linear unbiased predictor (EBLUP)
    • 1.5.3. Correlation Models
    • 1.5.4. Regression Models
    • 1.5.5. Implementation details
  • 1.6. Cross decomposition
  • 1.7. Naive Bayes
    • 1.7.1. Gaussian Naive Bayes
    • 1.7.2. Multinomial Naive Bayes
    • 1.7.3. Bernoulli Naive Bayes
    • 1.7.4. Out-of-core naive Bayes model fitting
  • 1.8. Decision Trees
    • 1.8.1. Classification
    • 1.8.2. Regression
    • 1.8.3. Multi-output problems
    • 1.8.4. Complexity
    • 1.8.5. Tips on practical use
    • 1.8.6. Tree algorithms: ID3, C4.5, C5.0 and CART
    • 1.8.7. Mathematical formulation
      • 1.8.7.1. Classification criteria
      • 1.8.7.2. Regression criteria
  • 1.9. Ensemble methods
    • 1.9.1. Forests of randomized trees
      • 1.9.1.1. Random Forests
      • 1.9.1.2. Extremely Randomized Trees
      • 1.9.1.3. Parameters
      • 1.9.1.4. Parallelization
      • 1.9.1.5. Feature importance evaluation
      • 1.9.1.6. Totally Random Trees Embedding
    • 1.9.2. AdaBoost
      • 1.9.2.1. Usage
    • 1.9.3. Gradient Tree Boosting
      • 1.9.3.1. Classification
      • 1.9.3.2. Regression
      • 1.9.3.3. Mathematical formulation
        • 1.9.3.3.1. Loss Functions
      • 1.9.3.4. Regularization
        • 1.9.3.4.1. Shrinkage
        • 1.9.3.4.2. Subsampling
      • 1.9.3.5. Interpretation
        • 1.9.3.5.1. Feature importance
        • 1.9.3.5.2. Partial dependence
  • 1.10. Multiclass and multilabel algorithms
    • 1.10.1. Multilabel classification format
    • 1.10.2. One-Vs-The-Rest
      • 1.10.2.1. Multiclass learning
      • 1.10.2.2. Multilabel learning
    • 1.10.3. One-Vs-One
      • 1.10.3.1. Multiclass learning
    • 1.10.4. Error-Correcting Output-Codes
      • 1.10.4.1. Multiclass learning
  • 1.11. Feature selection
    • 1.11.1. Univariate feature selection
    • 1.11.2. Recursive feature elimination
    • 1.11.3. L1-based feature selection
      • 1.11.3.1. Selecting non-zero coefficients
      • 1.11.3.2. Randomized sparse models
    • 1.11.4. Tree-based feature selection
    • 1.11.5. Feature selection as part of a pipeline
  • 1.12. Semi-Supervised
    • 1.12.1. Label Propagation
  • 1.13. Linear and quadratic discriminant analysis
    • 1.13.1. Dimensionality reduction using LDA
    • 1.13.2. Mathematical Idea
  • 1.14. Isotonic regression
• 2. Unsupervised learning
  • 2.1. Gaussian mixture models
    • 2.1.1. GMM classifier
      • 2.1.1.1. Pros and cons of class GMM: expectation-maximization inference
        • 2.1.1.1.1. Pros
        • 2.1.1.1.2. Cons
      • 2.1.1.2. Selecting the number of components in a classical GMM
      • 2.1.1.3. Estimation algorithm Expectation-maximization
    • 2.1.2. VBGMM classifier: variational Gaussian mixtures
      • 2.1.2.1. Pros and cons of class VBGMM: variational inference
        • 2.1.2.1.1. Pros
        • 2.1.2.1.2. Cons
      • 2.1.2.2. Estimation algorithm: variational inference
    • 2.1.3. DPGMM classifier: Infinite Gaussian mixtures
      • 2.1.3.1. Pros and cons of class DPGMM: Diriclet process mixture model
        • 2.1.3.1.1. Pros
        • 2.1.3.1.2. Cons
      • 2.1.3.2. The Dirichlet Process
  • 2.2. Manifold learning
    • 2.2.1. Introduction
    • 2.2.2. Isomap
      • 2.2.2.1. Complexity
    • 2.2.3. Locally Linear Embedding
      • 2.2.3.1. Complexity
    • 2.2.4. Modified Locally Linear Embedding
      • 2.2.4.1. Complexity
    • 2.2.5. Hessian Eigenmapping
      • 2.2.5.1. Complexity
    • 2.2.6. Spectral Embedding
      • 2.2.6.1. Complexity
    • 2.2.7. Local Tangent Space Alignment
      • 2.2.7.1. Complexity
    • 2.2.8. Multi-dimensional Scaling (MDS)
      • 2.2.8.1. Metric MDS
      • 2.2.8.2. Nonmetric MDS
    • 2.2.9. Tips on practical use
  • 2.3. Clustering
    • 2.3.1. Overview of clustering methods
    • 2.3.2. K-means
      • 2.3.2.1. Mini Batch K-Means
    • 2.3.3. Affinity Propagation
    • 2.3.4. Mean Shift
    • 2.3.5. Spectral clustering
      • 2.3.5.1. Different label assignment strategies
    • 2.3.6. Hierarchical clustering
      • 2.3.6.1. Adding connectivity constraints
    • 2.3.7. DBSCAN
    • 2.3.8. Clustering performance evaluation
      • 2.3.8.1. Inertia
        • 2.3.8.1.1. Presentation and usage
        • 2.3.8.1.2. Advantages
        • 2.3.8.1.3. Drawbacks
      • 2.3.8.2. Adjusted Rand index
        • 2.3.8.2.1. Presentation and usage
        • 2.3.8.2.2. Advantages
        • 2.3.8.2.3. Drawbacks
        • 2.3.8.2.4. Mathematical formulation
      • 2.3.8.3. Mutual Information based scores
        • 2.3.8.3.1. Presentation and usage
        • 2.3.8.3.2. Advantages
        • 2.3.8.3.3. Drawbacks
        • 2.3.8.3.4. Mathematical formulation
      • 2.3.8.4. Homogeneity, completeness and V-measure
        • 2.3.8.4.1. Presentation and usage
        • 2.3.8.4.2. Advantages
        • 2.3.8.4.3. Drawbacks
        • 2.3.8.4.4. Mathematical formulation
      • 2.3.8.5. Silhouette Coefficient
        • 2.3.8.5.1. Presentation and usage
        • 2.3.8.5.2. Advantages
        • 2.3.8.5.3. Drawbacks
  • 2.4. Biclustering
    • 2.4.1. Spectral Co-Clustering
      • 2.4.1.1. Mathematical formulation
    • 2.4.2. Spectral Biclustering
      • 2.4.2.1. Mathematical formulation
    • 2.4.3. Biclustering evaluation
  • 2.5. Decomposing signals in components (matrix factorization problems)
    • 2.5.1. Principal component analysis (PCA)
      • 2.5.1.1. Exact PCA and probabilistic interpretation
      • 2.5.1.2. Approximate PCA
      • 2.5.1.3. Kernel PCA
      • 2.5.1.4. Sparse principal components analysis (SparsePCA and MiniBatchSparsePCA)
    • 2.5.2. Truncated singular value decomposition and latent semantic analysis
    • 2.5.3. Dictionary Learning
      • 2.5.3.1. Sparse coding with a precomputed dictionary
      • 2.5.3.2. Generic dictionary learning
      • 2.5.3.3. Mini-batch dictionary learning
    • 2.5.4. Factor Analysis
    • 2.5.5. Independent component analysis (ICA)
    • 2.5.6. Non-negative matrix factorization (NMF or NNMF)
  • 2.6. Covariance estimation
    • 2.6.1. Empirical covariance
    • 2.6.2. Shrunk Covariance
      • 2.6.2.1. Basic shrinkage
      • 2.6.2.2. Ledoit-Wolf shrinkage
      • 2.6.2.3. Oracle Approximating Shrinkage
    • 2.6.3. Sparse inverse covariance
    • 2.6.4. Robust Covariance Estimation
      • 2.6.4.1. Minimum Covariance Determinant
  • 2.7. Novelty and Outlier Detection
    • 2.7.1. Novelty Detection
    • 2.7.2. Outlier Detection
      • 2.7.2.1. Fitting an elliptic envelop
      • 2.7.2.2. One-class SVM versus elliptic envelop
  • 2.8. Hidden Markov Models
    • 2.8.1. Using HMM
      • 2.8.1.1. Building HMM and generating samples
      • 2.8.1.2. Training HMM parameters and inferring the hidden states
      • 2.8.1.3. Implementing HMMs with custom emission probabilities
  • 2.9. Density Estimation
    • 2.9.1. Density Estimation: Histograms
    • 2.9.2. Kernel Density Estimation
  • 2.10. Neural network models (unsupervised)
    • 2.10.1. Restricted Boltzmann machines
      • 2.10.1.1. Graphical model and parametrization
      • 2.10.1.2. Bernoulli Restricted Boltzmann machines
      • 2.10.1.3. Stochastic Maximum Likelihood learning
• 3. Model selection and evaluation
  • 3.1. Cross-validation: evaluating estimator performance
    • 3.1.1. Computing cross-validated metrics
    • 3.1.2. Cross validation iterators
      • 3.1.2.1. K-fold
      • 3.1.2.2. Stratified k-fold
      • 3.1.2.3. Leave-One-Out - LOO
      • 3.1.2.4. Leave-P-Out - LPO
      • 3.1.2.5. Leave-One-Label-Out - LOLO
      • 3.1.2.6. Leave-P-Label-Out
      • 3.1.2.7. Random permutations cross-validation a.k.a. Shuffle & Split
      • 3.1.2.8. See also
      • 3.1.2.9. Bootstrapping cross-validation
    • 3.1.3. Cross validation and model selection
  • 3.2. Grid Search: Searching for estimator parameters
    • 3.2.1. Exhaustive Grid Search
      • 3.2.1.1. Scoring functions for parameter search
    • 3.2.2. Randomized Parameter Optimization
    • 3.2.3. Alternatives to brute force parameter search
      • 3.2.3.1. Model specific cross-validation
        • 3.2.3.1.1. sklearn.linear_model.RidgeCV
        • 3.2.3.1.2. sklearn.linear_model.RidgeClassifierCV
        • 3.2.3.1.3. sklearn.linear_model.LarsCV
        • 3.2.3.1.4. sklearn.linear_model.LassoLarsCV
        • 3.2.3.1.5. sklearn.linear_model.LassoCV
        • 3.2.3.1.6. sklearn.linear_model.ElasticNetCV
      • 3.2.3.2. Information Criterion
        • 3.2.3.2.1. sklearn.linear_model.LassoLarsIC
      • 3.2.3.3. Out of Bag Estimates
        • 3.2.3.3.1. sklearn.ensemble.RandomForestClassifier
        • 3.2.3.3.2. sklearn.ensemble.RandomForestRegressor
        • 3.2.3.3.3. sklearn.ensemble.ExtraTreesClassifier
        • 3.2.3.3.4. sklearn.ensemble.ExtraTreesRegressor
        • 3.2.3.3.5. sklearn.ensemble.GradientBoostingClassifier
        • 3.2.3.3.6. sklearn.ensemble.GradientBoostingRegressor
  • 3.3. Pipeline: chaining estimators
    • 3.3.1. Usage
    • 3.3.2. Notes
  • 3.4. FeatureUnion: Combining feature extractors
    • 3.4.1. Usage
  • 3.5. Model evaluation: quantifying the quality of predictions
    • 3.5.1. The scoring parameter: defining model evaluation rules
      • 3.5.1.1. Common cases: predefined values
      • 3.5.1.2. Defining your scoring strategy from score functions
      • 3.5.1.3. Implementing your own scoring object
    • 3.5.2. Function for prediction-error metrics
      • 3.5.2.1. Classification metrics
        • 3.5.2.1.1. Accuracy score
        • 3.5.2.1.2. Average precision score
        • 3.5.2.1.3. Confusion matrix
        • 3.5.2.1.4. Classification report
        • 3.5.2.1.5. Hamming loss
        • 3.5.2.1.6. Jaccard similarity coefficient score
        • 3.5.2.1.7. Precision, recall and F-measures
          • 3.5.2.1.7.1. Binary classification
          • 3.5.2.1.7.2. Multiclass and multilabel classification
        • 3.5.2.1.8. Hinge loss
        • 3.5.2.1.9. Log loss
        • 3.5.2.1.10. Matthews correlation coefficient
        • 3.5.2.1.11. Receiver operating characteristic (ROC)
        • 3.5.2.1.12. Zero one loss
      • 3.5.2.2. Regression metrics
        • 3.5.2.2.1. Explained variance score
        • 3.5.2.2.2. Mean absolute error
        • 3.5.2.2.3. Mean squared error
        • 3.5.2.2.4. R² score, the coefficient of determination
    • 3.5.3. Clustering metrics
    • 3.5.4. Biclustering metrics
      • 3.5.4.1. Clustering metrics
    • 3.5.5. Dummy estimators
• 4. Dataset transformations
  • 4.1. Feature extraction
    • 4.1.1. Loading features from dicts
    • 4.1.2. Feature hashing
      • 4.1.2.1. Implementation details
    • 4.1.3. Text feature extraction
      • 4.1.3.1. The Bag of Words representation
      • 4.1.3.2. Sparsity
      • 4.1.3.3. Common Vectorizer usage
      • 4.1.3.4. Tf–idf term weighting
      • 4.1.3.5. Decoding text files
      • 4.1.3.6. Applications and examples
      • 4.1.3.7. Limitations of the Bag of Words representation
      • 4.1.3.8. Vectorizing a large text corpus with the hashing trick
      • 4.1.3.9. Performing out-of-core scaling with HashingVectorizer
      • 4.1.3.10. Customizing the vectorizer classes
    • 4.1.4. Image feature extraction
      • 4.1.4.1. Patch extraction
      • 4.1.4.2. Connectivity graph of an image
  • 4.2. Preprocessing data
    • 4.2.1. Standardization, or mean removal and variance scaling
      • 4.2.1.1. Scaling features to a range
      • 4.2.1.2. Centering kernel matrices
    • 4.2.2. Normalization
    • 4.2.3. Binarization
      • 4.2.3.1. Feature binarization
    • 4.2.4. Encoding categorical features
    • 4.2.5. Label preprocessing
      • 4.2.5.1. Label binarization
      • 4.2.5.2. Label encoding
    • 4.2.6. Imputation of missing values
  • 4.3. Kernel Approximation
    • 4.3.1. Nystroem Method for Kernel Approximation
    • 4.3.2. Radial Basis Function Kernel
    • 4.3.3. Additive Chi Squared Kernel
    • 4.3.4. Skewed Chi Squared Kernel
    • 4.3.5. Mathematical Details
  • 4.4. Random Projection
    • 4.4.1. The Johnson-Lindenstrauss lemma
    • 4.4.2. Gaussian random projection
    • 4.4.3. Sparse random projection
  • 4.5. Pairwise metrics, Affinities and Kernels
    • 4.5.1. Cosine similarity
    • 4.5.2. Chi Squared Kernel
• 5. Dataset loading utilities
  • 5.1. General dataset API
  • 5.2. Toy datasets
  • 5.3. Sample images
  • 5.4. Sample generators
  • 5.5. Datasets in svmlight / libsvm format
  • 5.6. The Olivetti faces dataset
  • 5.7. The 20 newsgroups text dataset
    • 5.7.1. Usage
    • 5.7.2. Converting text to vectors
    • 5.7.3. Filtering text for more realistic training
  • 5.8. Downloading datasets from the mldata.org repository
  • 5.9. The Labeled Faces in the Wild face recognition dataset
    • 5.9.1. Usage
    • 5.9.2. Examples
  • 5.10. Forest covertypes