Agent Skills: Machine Learning

Machine Learning

Comprehensive machine learning skill covering the full ML lifecycle from experimentation to production deployment.

When to Use This Skill

• Building machine learning pipelines
• Feature engineering and data preprocessing
• Model training, evaluation, and selection
• Hyperparameter tuning and optimization
• Model deployment and serving
• ML experiment tracking and versioning
• Production ML monitoring and maintenance

ML Development Lifecycle

1. Problem Definition

Classification Types:

• Binary classification (spam/not spam)
• Multi-class classification (image categories)
• Multi-label classification (document tags)
• Regression (price prediction)
• Clustering (customer segmentation)
• Ranking (search results)
• Anomaly detection (fraud detection)

Success Metrics by Problem Type:

| Problem Type | Primary Metrics | Secondary Metrics | |--------------|-----------------|-------------------| | Binary Classification | AUC-ROC, F1 | Precision, Recall, PR-AUC | | Multi-class | Macro F1, Accuracy | Per-class metrics | | Regression | RMSE, MAE | R², MAPE | | Ranking | NDCG, MAP | MRR | | Clustering | Silhouette, Calinski-Harabasz | Davies-Bouldin |

2. Data Preparation

Data Quality Checks:

• Missing value analysis and imputation strategies
• Outlier detection and handling
• Data type validation
• Distribution analysis
• Target leakage detection

Feature Engineering Patterns:

• Numerical: scaling, binning, log transforms, polynomial features
• Categorical: one-hot, target encoding, frequency encoding, embeddings
• Temporal: lag features, rolling statistics, cyclical encoding
• Text: TF-IDF, word embeddings, transformer embeddings
• Geospatial: distance features, clustering, grid encoding

Train/Test Split Strategies:

• Random split (standard)
• Stratified split (imbalanced classes)
• Time-based split (temporal data)
• Group split (prevent data leakage)
• K-fold cross-validation

3. Model Selection

Algorithm Selection Guide:

| Data Size | Problem | Recommended Models | |-----------|---------|-------------------| | Small (<10K) | Classification | Logistic Regression, SVM, Random Forest | | Small (<10K) | Regression | Linear Regression, Ridge, SVR | | Medium (10K-1M) | Classification | XGBoost, LightGBM, Neural Networks | | Medium (10K-1M) | Regression | XGBoost, LightGBM, Neural Networks | | Large (>1M) | Any | Deep Learning, Distributed training | | Tabular | Any | Gradient Boosting (XGBoost, LightGBM, CatBoost) | | Images | Classification | CNN, ResNet, EfficientNet, Vision Transformers | | Text | NLP | Transformers (BERT, RoBERTa, GPT) | | Sequential | Time Series | LSTM, Transformer, Prophet |

4. Model Training

Hyperparameter Tuning:

• Grid Search: exhaustive, good for small spaces
• Random Search: efficient, good for large spaces
• Bayesian Optimization: smart exploration (Optuna, Hyperopt)
• Early stopping: prevent overfitting

Common Hyperparameters:

| Model | Key Parameters | |-------|---------------| | XGBoost | learning_rate, max_depth, n_estimators, subsample | | LightGBM | num_leaves, learning_rate, n_estimators, feature_fraction | | Random Forest | n_estimators, max_depth, min_samples_split | | Neural Networks | learning_rate, batch_size, layers, dropout |

5. Model Evaluation

Evaluation Best Practices:

• Always use held-out test set for final evaluation
• Use cross-validation during development
• Check for overfitting (train vs validation gap)
• Evaluate on multiple metrics
• Analyze errors qualitatively

Handling Imbalanced Data:

• Resampling: SMOTE, undersampling
• Class weights: weighted loss functions
• Threshold tuning: optimize decision threshold
• Evaluation: use PR-AUC over ROC-AUC

6. Production Deployment

Model Serving Patterns:

• REST API (Flask, FastAPI, TF Serving)
• Batch inference (scheduled jobs)
• Streaming (real-time predictions)
• Edge deployment (mobile, IoT)

Production Considerations:

• Latency requirements (p50, p95, p99)
• Throughput (requests per second)
• Model size and memory footprint
• Fallback strategies
• A/B testing framework

7. Monitoring & Maintenance

What to Monitor:

• Prediction latency
• Input feature distributions (data drift)
• Prediction distributions (concept drift)
• Model performance metrics
• Error rates and types

Retraining Triggers:

• Performance degradation below threshold
• Significant data drift detected
• Scheduled retraining (daily, weekly)
• New training data available

MLOps Best Practices

Experiment Tracking

Track for every experiment:

• Code version (git commit)
• Data version (hash or version ID)
• Hyperparameters
• Metrics (train, validation, test)
• Model artifacts
• Environment (packages, versions)

Model Versioning

models/
├── model_v1.0.0/
│   ├── model.pkl
│   ├── metadata.json
│   ├── requirements.txt
│   └── metrics.json
├── model_v1.1.0/
└── model_v2.0.0/

CI/CD for ML

1. Continuous Integration:

   • Data validation tests
   • Model training tests
   • Performance regression tests

2. Continuous Deployment:

   • Staging environment validation
   • Shadow mode testing
   • Gradual rollout (canary)
   • Automatic rollback

Reference Files

For detailed patterns and code examples, load reference files as needed:

• references/preprocessing.md - Data preprocessing patterns and feature engineering techniques
• references/model_patterns.md - Model architecture patterns and implementation examples
• references/evaluation.md - Comprehensive evaluation strategies and metrics

Integration with Other Skills

• performance - For optimizing inference latency
• testing - For ML-specific testing patterns
• database-optimization - For feature store queries
• debugging - For model debugging and error analysis