Scikit-learn Machine Learning
Industry-standard Python library for classical machine learning.
When to Use
- Classification or regression tasks
- Clustering or dimensionality reduction
- Preprocessing and feature engineering
- Model evaluation and cross-validation
- Hyperparameter tuning
- Building ML pipelines
Algorithm Selection
Classification
| Algorithm | Best For | Strengths | |-----------|----------|-----------| | Logistic Regression | Baseline, interpretable | Fast, probabilistic | | Random Forest | General purpose | Handles non-linear, feature importance | | Gradient Boosting | Best accuracy | State-of-art for tabular | | SVM | High-dimensional data | Works well with few samples | | KNN | Simple problems | No training, instance-based |
Regression
| Algorithm | Best For | Notes | |-----------|----------|-------| | Linear Regression | Baseline | Interpretable coefficients | | Ridge/Lasso | Regularization needed | L2 vs L1 penalty | | Random Forest | Non-linear relationships | Robust to outliers | | Gradient Boosting | Best accuracy | XGBoost, LightGBM wrappers |
Clustering
| Algorithm | Best For | Key Parameter | |-----------|----------|---------------| | KMeans | Spherical clusters | n_clusters (must specify) | | DBSCAN | Arbitrary shapes | eps (density) | | Agglomerative | Hierarchical | n_clusters or distance threshold | | Gaussian Mixture | Soft clustering | n_components |
Dimensionality Reduction
| Method | Preserves | Use Case | |--------|-----------|----------| | PCA | Global variance | Feature reduction | | t-SNE | Local structure | 2D/3D visualization | | UMAP | Both local/global | Visualization + downstream |
Pipeline Concepts
Key concept: Pipelines prevent data leakage by ensuring transformations are fit only on training data.
| Component | Purpose | |-----------|---------| | Pipeline | Sequential steps (transform → model) | | ColumnTransformer | Apply different transforms to different columns | | FeatureUnion | Combine multiple feature extraction methods |
Common preprocessing flow:
- Impute missing values (SimpleImputer)
- Scale numeric features (StandardScaler, MinMaxScaler)
- Encode categoricals (OneHotEncoder, OrdinalEncoder)
- Optional: feature selection or polynomial features
Model Evaluation
Cross-Validation Strategies
| Strategy | Use Case | |----------|----------| | KFold | General purpose | | StratifiedKFold | Imbalanced classification | | TimeSeriesSplit | Temporal data | | LeaveOneOut | Very small datasets |
Metrics
| Task | Metric | When to Use | |------|--------|-------------| | Classification | Accuracy | Balanced classes | | | F1-score | Imbalanced classes | | | ROC-AUC | Ranking, threshold tuning | | | Precision/Recall | Domain-specific costs | | Regression | RMSE | Penalize large errors | | | MAE | Robust to outliers | | | R² | Explained variance |
Hyperparameter Tuning
| Method | Pros | Cons | |--------|------|------| | GridSearchCV | Exhaustive | Slow for many params | | RandomizedSearchCV | Faster | May miss optimal | | HalvingGridSearchCV | Efficient | Requires sklearn 0.24+ |
Key concept: Always tune on validation set, evaluate final model on held-out test set.
Best Practices
| Practice | Why | |----------|-----| | Split data first | Prevent leakage | | Use pipelines | Reproducible, no leakage | | Scale for distance-based | KNN, SVM, PCA need scaled features | | Stratify imbalanced | Preserve class distribution | | Cross-validate | Reliable performance estimates | | Check learning curves | Diagnose over/underfitting |
Common Pitfalls
| Pitfall | Solution |
|---------|----------|
| Fitting scaler on all data | Use pipeline or fit only on train |
| Using accuracy for imbalanced | Use F1, ROC-AUC, or balanced accuracy |
| Too many hyperparameters | Start simple, add complexity |
| Ignoring feature importance | Use feature_importances_ or permutation importance |
Resources
- Docs: https://scikit-learn.org/
- User Guide: https://scikit-learn.org/stable/user_guide.html
- Algorithm Cheat Sheet: https://scikit-learn.org/stable/tutorial/machine_learning_map/