Search & Replay Engine

In Icho v0.5.0, cassettes are no longer just static mock files—they represent Searchable AI Executions.

The Reproducible AI Paradigm

Yesterday, a user reported a bug: "The AI gave the wrong answer."

The problem? Without recording the exact prompt, model version, tool calls, and execution context, reproducing the issue becomes harder than fixing it. Git lets us go back to any commit; logs let us inspect what happened. Icho brings that exact reproducibility to AI executions.

1. CLI Search (`icho search`)

Use zero-dependency TF-IDF vector embeddings and Cosine Similarity to find past executions instantly by prompt text, response content, time range, provider, or model:

$ icho search "refund" --since yesterday --provider groq --top-k 5 -i

Supported Search Flags:

query: Semantic search query scored against input messages and response text using TF-IDF cosine similarity.
--since / -s: Filter executions created since relative time (yesterday, today, 10m, 2h, 1d, 7d) or ISO datetime string.
--until / -u: Filter executions created prior to specified relative time or ISO timestamp.
--provider: Filter by LLM provider name (e.g., groq, openai, anthropic).
--model / -m: Filter by model identifier (e.g., llama-3.1-8b-instant).
--interactive / -i: Interactively select a candidate search result to inspect details and print ready-to-use Python replay code.

2. CLI Replay (`icho replay`)

Inspect any cassette by ID, hash, or filename and generate local Python replay snippets to re-run the exact context in your test suite:

$ icho replay a1f7d9e4

3. Execution Log (`icho log`)

View a git-like chronological log of recorded AI executions complete with input/output snippets, latency metrics, and model names:

$ icho log -n 10

4. Python API Search (`search_cassettes`)

Perform programmatic search directly within your Python test suites, evaluation loops, or debugging scripts:

from icho.search import search_cassettes

# Query executions recorded since yesterday
results = search_cassettes( 
    query="refund request",
    path="tests/cassettes",
    since="yesterday",
    top_k=5
)

for res in results:
    print(f"[{res.score:.4f}] {res.hash} - {res.input_snippet}")

Return to Icho Home