google/lightflow

Benchmarking Lightflow Workflows

Use benchmarks/run_benchmark.py to measure and verify Arm A (Lightflow DAG) against Arm B (Steelmanned Traditional Skill: benchmarks/baseline_skills/<name>/SKILL.md + examples/<name>/actions.py) across all 7 domain workflow pairs (hn_digest, usgs_seismic_alert, pypi_upgrade_guard, async_job_watcher, blue_green_release, incident_db_failover, tenant_gitops_onboarding) plus the create_lightflow authoring workflow.

---

Commands

# Run the unified A/B benchmark (Markdown report)
python3 benchmarks/run_benchmark.py

# Run N=5 independent trials to verify multi-run consistency
python3 benchmarks/run_benchmark.py --trials=5

# Emit machine-readable JSON metrics
python3 benchmarks/run_benchmark.py --trials=5 --json

---

What Is Verified on Every Run

1. Ground-Truth Correctness & Invariants:
   • order_match: Completed stages in passport.json match the exact expected topological sequence.
   • no_duplicate_side_effects: Every stage completes at most once (1x), including across mid-run failures and lightflow resume recoveries (pypi_upgrade_guard, blue_green_release, incident_db_failover).
   • gate_discipline: Every operator_action stage records a PAUSED stamp (exit 2) before its COMPLETED stamp.
   • external_state_valid & baseline_external_state_valid: Both Arm A and Arm B produce the exact expected on-disk artifacts, intermediate rollbacks, and cleanups.
2. Context & Code Generation Cost (1 token ≈ 4 chars):
   • Skill Loaded: Universal skills/run_lightflows/SKILL.md (~1,097 tok once per session) vs. per-workflow benchmarks/baseline_skills/<name>/SKILL.md (~7,075 tok across the 7 domain workflows).
   • Agent Command Output (cmd): Compact declarative lightflow start / resume CLI calls (~795 tok across 7 workflows) vs. inline python3 -c orchestration snippets (~2,992 tok across 7 workflows).
   • Out-of-Context State: passport.json bytes persisted on disk (52,958 chars / ~13,240 tok) and lightflow.yaml + actions.py source bytes avoided (115,321 chars / ~28,830 tok, 83% warm zero-context savings).