What
SIA (Self Improving AI) implements a 3-agent loop — Meta, Target, and Feedback — for autonomous benchmark improvement. The Meta agent designs task-specific agents, the Target agent executes tasks, and the Feedback agent reviews performance and drives improvement. Results include 56.6% LawBench gain, #1 on MLE-Bench, and 14x GPU kernel speedup. The three agent roles map to b00t patterns: Meta ≈ research-soul, Feedback ≈ lfmf + discriminator, and generations ≈ checkpoints.
Installation is via pip install 'sia-agent[claude]' or pip install 'sia-agent[openhands]'. Run with sia run --task gpqa --max_gen 5 --run_id 1. Visualize results with sia web. Custom tasks use sia run --task_dir ./my-task --max_gen 5 --run_id 1. MLE-Bench requires dataset preparation via python -m sia.prepare_mlebench_dataset -c 'spaceship-titanic'.
When to Use
Use SIA for autonomous task improvement, benchmark optimization, and any scenario requiring a feedback-driven self-improvement loop. It is particularly effective for ML benchmarks and agent optimization tasks.
How
- Install:
pip install 'sia-agent[claude]' - Run a task:
sia run --task gpqa --max_gen 5 --run_id 1 - Visualize results:
sia web - For custom tasks:
sia run --task_dir ./my-task --max_gen 5 --run_id 1 - For MLE-Bench: prepare dataset with
python -m sia.prepare_mlebench_dataset -c 'spaceship-titanic'