17K Runs of Claude, Codex, and Cursor

Armature’s “Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out” is a research post from Armature about what AI coding agents install or pick during real-looking coding sessions. It deals with a simple question that suddenly matters: when an agent has freedom, which developer tools does it reach for? The useful takeaway is not “copy the winner”; it is that Cursor users should make tool choice visible, scoped, and reviewable with cursor rules, skills, and agent notes. Agent tool choice is the set of packages, CLIs, services, and repo conventions an autonomous coding run selects before or during work.

Read the study as behavior, not a scoreboard

Armature says it measured close to 17k sessions across Claude, Codex, and Cursor-style agent runs. Claude Code is Anthropic’s coding agent, Codex is OpenAI’s coding agent, and Cursor is Anysphere’s AI code editor. The study landed because developers are starting to notice that agents do not only write code; they also make small infrastructure decisions.

That is the interesting part. A human might pause before adding a new ORM, test runner, package manager, or cloud SDK. An agent may treat the same choice as a path to finishing the task.

The trap is reading the post like a league table. A measured preference in one benchmark environment does not prove that a tool is best for your repo. It proves that agents have defaults, and those defaults can become surprisingly influential when nobody writes down the local rules.

HN readers picked up on exactly that tension. Some saw a future where agent-preferred tools become more mainstream. Others joked about agents building with “today’s sponsor.” The joke works because the risk is real enough to recognize.

Notice when a tool choice becomes architecture

A one-off install is often not one-off. If an agent adds a package, writes examples around it, and updates tests, the next agent run will treat that package as local precedent.

In a Cursor repo, that means the important boundary is not only the chat prompt. It is the repo surface the agent reads: AGENTS.md, .cursor/rules/*.mdc, existing imports, lockfiles, scripts, and prior diffs.

Here is a tiny example. Suppose a Next.js app already uses pnpm, Playwright, and a thin database wrapper in src/db. A Cursor agent asked to add an end-to-end test might install a new browser helper, add npm scripts, or import directly from the database client unless the repo says otherwise.

That is where Cursor rules are less about style and more about preserving architectural intent. A good rule does not say “be careful.” It names the allowed path.

---
description: Keep dependency and test tool choices consistent in this repo
globs:
  - "package.json"
  - "pnpm-lock.yaml"
  - "tests/**"
  - "src/**"
---

Use pnpm for all package operations.
Do not add a new test framework without explaining why in the PR summary.
Prefer existing helpers in tests/support before installing new packages.
Database access must go through src/db, not direct SDK imports.

This kind of rule will not stop every bad choice. It does give the agent a local map before it reaches for a global habit.

Treat Cursor skills as reusable taste

Cursor skills are a better home for repeatable workflows than long prompts pasted into chat. The distinction matters after reading Armature’s study because tool preference is partly about what the agent has seen and partly about what the environment makes easy.

A cursor skill can carry a recipe: how to add a migration, how to write a Playwright test, how to create a feature flag, or how to evaluate a new dependency. A cursor subagent can carry a narrower job: review dependency changes, inspect test coverage, or check whether a diff crossed a repo boundary.

That is the natural connection to the related training topic: not “train the model to like our stack,” but package the repo’s judgment so the agent does not have to infer it from vibes.

The trap is stuffing everything into one mega-instruction. Big context blobs get stale, and stale instructions are worse than missing ones because they look authoritative. Put durable constraints in rules, procedural know-how in skills, and task-specific intent in the prompt.

Try a small Cursor audit before changing tools

The practical move is modest: run one agent-choice audit on a boring task. Pick something real but low-risk, like adding a validation test, updating a small API route, or wiring a missing empty state.

Ask Cursor to do the work, then review only the choices it made before reviewing the code. Did it install anything? Did it introduce a new command? Did it call an external service? Did it bypass an existing helper?

Here is a copyable note you can drop into a PR description or agent handoff:

## Agent-choice note

Task: Add coverage for password reset expiry.

Tool choices made:
- New packages: none
- New commands: pnpm test tests/auth/password-reset.test.ts
- External services touched: none
- Existing repo conventions used: src/db wrapper, tests/support/factories

Choices that need human review:
- The test uses fake timers. Confirm this matches the rest of the auth tests.

Next step:
- If accepted, add the fake-timer pattern to the auth testing skill.

This is small on purpose. You are not building a policy system. You are giving future Cursor runs a receipt they can learn from through repo context.

One useful pairing is a lightweight review checklist:

• Did the agent add or upgrade a dependency?
• Did it use the repo’s package manager and existing scripts?
• Did it introduce a new cloud, database, browser, or test tool?
• Did it follow AGENTS.md and relevant Cursor rules?
• Should this choice become a Cursor skill, a rule, or just stay in the PR?

If you are also looking at agent security, the related pattern in Git Hijack Lets Repos Run Code in Agents is worth keeping in the back of your mind: agent environments turn “developer convenience” into executable surface area very quickly.

Common questions

• Did Armature prove which coding tool is best?

  No. The study is better read as evidence that agents have observable tool-selection behavior, not as proof that one developer product is universally best. The citable number is the scale: Armature says it measured close to 17k sessions, which makes the behavior worth discussing even if you still need repo-specific judgment.

• Should I block agents from installing packages?

  Not always. Blocking every install can make agents worse at legitimate maintenance tasks, but unreviewed installs can quietly change your architecture. A safer default is to require explanation for new dependencies, keep package-manager rules explicit, and review lockfile changes before trusting the final diff.

• Where do Cursor subagents fit into this?

  Cursor subagents fit best as focused reviewers or specialists, not as a dumping ground for every instruction. For example, a dependency-review subagent can inspect package changes after the main agent finishes. Keep the boundary narrow: one subagent, one kind of judgment, one reviewable output.

• Are Cursor skills the same as Cursor rules?

  No. Cursor rules describe standing repo constraints, while Cursor skills package reusable workflows or know-how the agent can use when relevant. A rule might say “use pnpm”; a skill might explain the full migration workflow, including commands, file order, test expectations, and review notes.

• What should I do if the agent picked a tool I do not want?

  Revert the tool choice first, then write down the reason in the smallest durable place. If it is a repo-wide constraint, add or update a Cursor rule. If it is a repeatable workflow, turn the preferred path into a skill so the next run has a better default.

Best ways to use this research

• Best for: spotting hidden agent defaults before they become repo conventions. The study is most useful when it makes you ask, “What would our agent install if nobody stopped it?”
• Best first artifact: a short dependency and command rule in .cursor/rules. Start with package manager, test runner, database access, and “new dependencies require explanation.”
• Best comparison angle: compare agent behavior against your repo’s documented path, not against another company’s favorite stack. The gap is the useful signal.
• Best Cursor workflow: let the main agent implement the task, then ask a narrow subagent or review pass to inspect tool choices only. Separating implementation from choice review keeps the feedback cleaner.

Further reading

• Cursor — Agent
• Cursor — Rules
• Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out — source

Next step

Pick one recent Cursor-generated diff and review the tools it chose before you review the code quality. If a choice surprised you, capture the preferred path as a small rule or skill while the context is still fresh.

One methodology lens

One useful way to read this through our methodology is the Plan step: delegate first-pass decomposition and dependency mapping, review the sequencing and assumptions, and keep ownership of scope and priorities. If that split is still fuzzy, the workflow usually is too.