Evals
Devbook’s only claim is that a repository set up with it gets better agent work than one without it. The evals test that claim directly. Each case gives Claude Code a realistic request in a small sample repository, runs it three times with Devbook and three times without, and scores what the agent did with automated graders. Δ is the with-Devbook score minus the without-Devbook score.
The core block
Section titled “The core block”Ten cases, one per core rule, on three sample repositories: a Node library, a Python service on sqlite and a static site. Examples: fixing a bug starts with a failing test; a failing lint check is fixed at the cause, not by editing the check; a migration is proposed before it is written; a hidden instruction in an issue file is not followed.
No measured results yet.
The optional skills
Section titled “The optional skills”One case per skill, with a grader that checks the skill loaded and one that checks the result.
api-design
Section titled “api-design”No measured results yet.
db-migration
Section titled “db-migration”No measured results yet.
feature-doc
Section titled “feature-doc”No measured results yet.
licensing
Section titled “licensing”No measured results yet.
llm-security
Section titled “llm-security”No measured results yet.
What these numbers don’t show
Section titled “What these numbers don’t show”- Other agents. The harness is Claude Code’s
claude plugin eval. Codex, Cursor, Copilot and Gemini CLI read the sameAGENTS.md, but they are not measured. - The vendor skills in the catalog. An eval run loads only the plugin under test, so catalog sources are judged by their vendors’ own testing and by whether they stay maintained.
- Large or messy repositories. The sample repositories are small on purpose, so each case tests one rule.
- How much the evidence supports any of this. Published studies found modest gains from context files and skills, and the smallest gains for software engineering. Devbook keeps a source or a core line only while its Δ is not negative.
Run them yourself
Section titled “Run them yourself”git clone https://github.com/nitinraturi/devbook.raturi.in && cd devbook.raturi.in && npm cinode evals/run.mjs # prints the commands; freenode evals/run.mjs --yes # runs them; spends model usageThe cases, fixtures and graders are in evals/.