caliper

项目详情摘要

Caliper: Know if your agent skill actually works Caliper is a lightweight evaluation harness for agent skills. Write a short spec of what "good" looks like, run it, and get a **success rate** you can track. Works with the agent you already use: **Claude Code, Codex, Pi, or Hermes**. Caliper installs the skill where the agent looks for skills and lets the agent choose. Teach your agent to evaluate:** Or run it yourself:** You write a spec, a YAML file describing what "working" means. Either hand-write it or have `/grill-skill` generate it for you. `--ablate` runs the same tasks with that skill *removed* — a declared MCP server can be ablated the same way — and `caliper compare` diffs the two runs task by task: Agent skills are hard to test. A skill that works on your machine, on this prompt, today, might fail tomorrow after a model update or a one-line prompt edit. Caliper makes reliability measurable: define what success looks like, run the skill repeatedly, and get a success rate you can track over time. Use Caliper to answer questions like: Is my agent still working the same with this new model? Did my prompt edit improved the skill? Does my skill fire when it should, and stay quiet when it needs to not trigger? Is the skill worth the context? Or would the base agent pass without it? Does it still pass the workflows it passed last week? Which agent (Claude Code, Codex, Pi, or Hermes) runs this skill more reliably? Quick start Path A: Agentic (let your agent drive) 1. Install the skills** 2. Generate a spec interactively** In your agent (Claude Code or Codex): `grill-skill` reads your `SKILL.md`, interviews you, and writes a 3-task `.eval.yaml` (happy path, edge case, adversarial). 3. Run and measure** Browse past runs: Path B: CLI (run it yourself) 1. Install the CLI*