lab
An automation agent with no irreversible side effects
Can an AI workflow stay useful without executing irreversible external actions?
Hypothesis: Suggestion / assisted modes, with a timeline and feedback, are enough for a credible MVP. Full autonomy is not the first deliverable.
Setup
Public repo AI Automation Agent, commit 19343e5. Same gold set as hitl-agent.v1: data/labeval/hitl-agent.v1.cases.json.
Measured fixes: word-boundary keywords (merci ≠ commerciale), mixed-intent priority (support > admin > commercial > reporting), low_risk_auto reserved for reporting.
Reproduce: python -m labeval --run hitl-agent.
Result
Run hitl-agent.v2 above, versus v1 on the same 20 cases:
| | v1 (e920cf9) | v2 (19343e5) |
| --- | --- | --- |
| Classification | 85% | 100% |
| Extraction | 90% | 96.25% |
| Tool choice | 95% | 100% |
| False autonomy | 60% | 15% |
| Human review | 35% | 80% |
The three remaining false-autonomy cases are admin / incident still on assisted while gold asks for suggestion_only.
Failure / limit
Heuristic path, not a live LLM, not Gmail/Slack. The v1 snapshot stays published: that is the before. MCP is out of this run.
Next
The same gold with provider_live when a key is available.