# Evals

> How Devbook measures its own effect - each case run with and without Devbook in Claude Code, scored by automated graders - and the latest results.

Devbook's only claim is that a repository set up with it gets better agent work than one without it. The evals test that claim directly. Each case gives Claude Code a realistic request in a small sample repository, runs it three times with Devbook and three times without, and scores what the agent did with automated graders. Δ is the with-Devbook score minus the without-Devbook score.

## The core block

Ten cases, one per core rule, on three sample repositories: a Node library, a Python service on sqlite and a static site. Examples: fixing a bug starts with a failing test; a failing lint check is fixed at the cause, not by editing the check; a migration is proposed before it is written; a hidden instruction in an issue file is not followed.

No measured results yet.

## The optional skills

One case per skill, with a grader that checks the skill loaded and one that checks the result.

### api-design

No measured results yet.

### db-migration

No measured results yet.

### feature-doc

No measured results yet.

### licensing

No measured results yet.

### llm-security

No measured results yet.

## What these numbers don't show

- **Other agents.** The harness is Claude Code's `claude plugin eval`. Codex, Cursor, Copilot and Gemini CLI read the same `AGENTS.md`, but they are not measured.
- **The vendor skills in the catalog.** An eval run loads only the plugin under test, so catalog sources are judged by their vendors' own testing and by whether they stay maintained.
- **Large or messy repositories.** The sample repositories are small on purpose, so each case tests one rule.
- **How much the evidence supports any of this.** Published studies found modest gains from context files and skills, and the smallest gains for software engineering. Devbook keeps a source or a core line only while its Δ is not negative.

## Run them yourself

```sh
git clone https://github.com/nitinraturi/devbook.raturi.in && cd devbook.raturi.in && npm ci
node evals/run.mjs          # prints the commands; free
node evals/run.mjs --yes    # runs them; spends model usage
```

The cases, fixtures and graders are in [`evals/`](https://github.com/nitinraturi/devbook.raturi.in/tree/main/evals).