Experiment Tracking
Track ML experiments, metrics, and models.
Comparison
| Platform | Best For | Self-hosted | Visualization | |----------|----------|-------------|---------------| | MLflow | Open-source, model registry | Yes | Basic | | W&B | Collaboration, sweeps | Limited | Excellent | | Neptune | Team collaboration | No | Good | | ClearML | Full MLOps | Yes | Good |
MLflow
Open-source platform from Databricks.
Core components:
- Tracking: Log parameters, metrics, artifacts
- Projects: Reproducible runs (MLproject file)
- Models: Package and deploy models
- Registry: Model versioning and staging
Strengths: Self-hosted, open-source, model registry, framework integrations Limitations: Basic visualization, less collaborative features
Key concept: Autologging for major frameworks - automatic metric capture with one line.
Weights & Biases (W&B)
Cloud-first experiment tracking with excellent visualization.
Core features:
- Experiment tracking: Metrics, hyperparameters, system stats
- Sweeps: Hyperparameter search (grid, random, Bayesian)
- Artifacts: Dataset and model versioning
- Reports: Shareable documentation
Strengths: Beautiful visualizations, team collaboration, hyperparameter sweeps Limitations: Cloud-dependent, limited self-hosting
Key concept: wandb.init() + wandb.log() - simple API, powerful features.
What to Track
| Category | Examples | |----------|----------| | Hyperparameters | Learning rate, batch size, architecture | | Metrics | Loss, accuracy, F1, per-epoch values | | Artifacts | Model checkpoints, configs, datasets | | System | GPU usage, memory, runtime | | Code | Git commit, diff, requirements |
Model Registry Concepts
| Stage | Purpose | |-------|---------| | None | Just logged, not registered | | Staging | Testing, validation | | Production | Serving live traffic | | Archived | Deprecated, kept for reference |
Decision Guide
| Scenario | Recommendation | |----------|----------------| | Self-hosted requirement | MLflow | | Team collaboration | W&B | | Model registry focus | MLflow | | Hyperparameter sweeps | W&B | | Beautiful dashboards | W&B | | Full MLOps pipeline | MLflow + deployment tools |
Resources
- MLflow: https://mlflow.org/docs/latest/
- W&B: https://docs.wandb.ai/