Snapshot Testing & Replay for LLM Applications
Turn LLM executions into searchable, reproducible cassettes. Search past requests ($ icho search "refund" --since yesterday) and replay them locally in 12ms to debug and validate fixes.
# Icho Snapshot Cassette Testing
from langchain_groq import ChatGroq
from icho import cassette
model = ChatGroq(model_name="llama-3.1-8b-instant")
# Automatically intercepts, records or replays
with cassette("tests/cassettes"):
response = model.invoke("Write a 3-word slogan for gravity.")
🟢⏱️ First run: 2.3 seconds (Recorded to cassette)
🚀⏱️ Second run: 12 ms (Replayed locally!)
See the Record & Replay Cycle
Interactive visualization showing what happens on the first run (recording network responses) versus subsequent runs (instant replaying).
Replay vs. Network Comparison
Searchable AI Executions & Instant Replay
Icho is not just an API snapshot decorator; it turns past AI executions into a searchable vector index. Query by prompt text, filter by time (--since yesterday), and instantly replay locally to reproduce bugs.
================================================================================
🧪 Icho AI Dataset Test Suite Runner (v0.8.0)
Target Domain: tests/customer_support/ (Found 3 cassette files)
================================================================================
✓ greeting.yaml [PASSED] 12ms (Match score: 1.00 - Replayed)
✓ refund.yaml [PASSED] 14ms (Match score: 0.98 - Replayed)
✓ complaint.yaml [PASSED] 11ms (Match score: 0.94 - Replayed)
--------------------------------------------------------------------------------
Suite Summary: 3/3 PASSED | 0 Failed | Total Time: 37ms | Drift: 0.00%
✨ Parallel Worker Pool (-j 4) active • Zero API Costs Accrued