Overview
Icho is a lightweight snapshot testing and execution replay library for LLM applications. It intercepts, records, searches, and replays HTTP request payloads, making AI applications reproducible and turning test suites fast and deterministic.
New in Icho v0.8.0: AI Test Datasets (icho test) 🧪
"Organize thousands of conversation scenarios into domain directories using .yaml or .json cassettes, run parallel worker pools, and detect prompt regression automatically."
Run batch suites across domain datasets (icho test tests/customer_support/) or run multi-domain datasets with parallel workers (icho test tests/ -j 4).
Traditional LLM tests suffer from three main friction points:
- Flakiness: LLM APIs have non-deterministic outputs which can break test expectations randomly.
- Speed: Requests to external LLM providers take anywhere from 1.5s to 5s, ballooning test suite durations.
- Cost: Repeatedly calling commercial APIs in CI/CD environments accumulates significant financial overhead.