The Power Behind Icho
Unlock reproducible AI applications. Batch run AI test datasets (`icho test`), search past executions with vector similarity, auto-mask PII data, replay stream generators, and eliminate API latency.
Engineered for LLM Reliability
A lightweight, zero-dependency Python utility designed to make LLM applications deterministic and rock-solid during development.
AI Test Datasets & Batch Suites (v0.8.0)
Organize thousands of conversation scenarios into domain directories using .yaml or .json cassettes. Run batch test suites ('icho test tests/'), parallel worker pools (-j 4), and regression testing (--mode regression).
Search & Replay Engine
Search past AI executions with TF-IDF cosine vector similarity, filter by timestamps ('--since yesterday'), and generate exact local Python replay code to reproduce bugs.
Flexible Storage Spaces
Support for File, Memory, and PostgreSQL backends (storage='file', 'memory', 'postgres' or StorageBackend instances).
Multiple Execution Modes
Supports auto, record, replay, and live modes. Swap effortlessly between local testing, CI build pipelines, or raw live access.
PII & Sensitive Data Masking
Automatically redacts emails, phone numbers, credit cards, SSNs, IP addresses, API keys, and bearer tokens from stored cassettes.
NVIDIA NeMo Guardrails
Apply official NVIDIA NeMo Guardrails (nemoguardrails) on input prompts before LLM execution and output responses after generation.
Tool Calling & Functions
Full support for recording, hashing, and replaying tool call requests, function definitions, and response histories deterministically.
Streaming & Async Generators
Full recording and playback for streaming responses (both sync and async generators) across OpenAI, Anthropic, and LangChain.
Structured Output Support
Natively record and replay schema-conforming structured outputs generated via LangChain's .with_structured_output() mechanism.
Multi-Provider Adapters
Built-in automatic monkeypatching for OpenAI, Anthropic, and LangChain Groq. No manual network hooks required.
Metadata Index Generation
Automatically builds and updates a metadata.json catalog in cassette directories for fast stats, inspection, and visualization.
Deterministic Hashing
Recursively sorts request inputs and prompt bodies before hashing to ensure consistent cassette lookup despite param ordering.
Ignored Dynamic Fields
Strip dynamic or unstable fields (e.g. temperature, max_tokens, dynamic prompts) to prevent unwanted cache invalidation.
Custom Normalizers
Run user-defined functions to redact sensitive content, strip timestamps, or sanitize request payloads before saving.
Built to Fit Your CI Pipelines
Icho integrates seamlessly into Python testing frameworks like `pytest` and runs efficiently inside GitHub Actions, GitLab CI, or local Docker environments.
Searchable AI Executions & Replay
Find past executions by prompt cosine similarity (icho search "refund" --since yesterday) and replay them locally in 12ms.
Flexible Storage Spaces (File, Memory, Postgres)
Choose between disk JSON files, zero-disk in-memory storage for unit tests, or central PostgreSQL database storage.
Automatic PII & Secret Redaction
Enable mask_pii=True to automatically redact email addresses, phone numbers, credit card numbers, and API keys.
# Icho v0.8.0: AI Test Datasets (icho test) 🧪
from icho import discover_suite, SuiteRunner
# 1. Discover nested test suite hierarchy (.yaml or .json cassettes)
suite = discover_suite("tests/customer_support")
# 2. Execute batch test suite across domain datasets
runner = SuiteRunner(mode="replay")
report = runner.run(suite)
print(report.render_text())
CLI: $ icho test tests/customer_support/ --mode regression --threshold 0.85 --fail-on-drift