AI Test Datasets (`icho test`) 🧪

In Icho v0.8.0, organize thousands of conversation scenarios into domain directories using .yaml or .json cassettes. Execute batch test suites, parallel worker pools, and regression suites against reference cassettes.

📁 Recommended Directory Structure:

tests/
├── customer_support/
│   ├── greeting.yaml
│   ├── refund.yaml
│   └── complaint.yaml
├── travel_agent/
│   ├── booking.yaml
│   └── cancellation.yaml
└── finance/
    └── invoices.yaml

1. CLI Commands (`icho test`)

# Run batch test suite across a domain directory:

icho test tests/customer_support/

# Run multi-domain dataset with parallel worker pool:

icho test tests/ -j 4

# Run regression suite across datasets against reference cassettes:

icho test tests/customer_support/ --mode regression --threshold 0.85 --fail-on-drift

2. Python API (`discover_suite`, `SuiteRunner`)

Programmatically discover and run nested test suite hierarchies in Python:

from icho import discover_suite, SuiteRunner

# Discover nested test suite hierarchy
suite = discover_suite("tests/customer_support")

# Execute batch test suite
runner = SuiteRunner(mode="replay")
report = runner.run(suite)

print(report.render_text())

Return to Icho Home