Oqoqo Measures Agents on Real Tasks

Oqoqo is an eval and custom benchmark builder from the Oqoqo team, posted as a Show HN project in August 2026. It deals with a plain problem: benchmark scores from clean labs often do not predict whether an agent can use your real product surface, CLI, MCP server, or code review flow. The useful takeaway for Cursor users is to test the work you actually ask agents to do, including a cursor code review, instead of trusting a generic leaderboard. An agent eval is a repeatable task with a starting state, instructions, evidence, and a pass/fail judgment that can be rerun after product or prompt changes.

A cursor ai code editor review can tell you how an IDE feels; Oqoqo's angle is whether an agent completes the same job reliably. That distinction is why developers cared about the Show HN post. The pitch was not another abstract model scoreboard. It was a way to measure agent-facing interfaces in the messy places where agents actually operate.

Understand what Oqoqo is measuring

Oqoqo is aimed at realistic evals and custom benchmarks for tasks users care about. In the Show HN description, the authors say it can measure how agent-friendly product surfaces are across coding agents and IDE agents, including Cursor, Anysphere's AI code editor, Claude Code, Anthropic's coding agent, and GitHub Copilot, GitHub's AI coding assistant.

That matters because the agent surface is now part of the product. A documentation page, CLI flag, MCP tool response, or repo rule can be clear to a human and still confusing to an agent. Oqoqo's promise is to turn that fuzziness into something you can rerun.

The trap is treating this as a universal ranking. As of August 2026, the useful claim is narrower: Oqoqo is about making your own work reproducible. A benchmark that proves your billing CLI is easier for an agent to use does not prove your whole stack is agent-friendly.

Separate real signal from leaderboard comfort

The strongest part of the Oqoqo idea is regression testing. The authors explicitly call out MCP, CLI, skills, SDKs, and other agent-facing interfaces. Those are exactly the surfaces that change quietly and break agent workflows in boring ways.

Take a real example. Suppose your repo has a packages/billing service, a Cursor rule for security-sensitive changes, and an MCP server that exposes invoice fixtures. A useful eval is not: can an agent solve billing bugs in general? It is: can the agent inspect a tax rounding change, read the right rule, call the read-only fixture tool, and leave a review with evidence?

That result does not prove the agent is smart. It proves the path is navigable. For Cursor users, that is often the more valuable finding, because cursor rules, cursor skills, and cursor subagents can improve the path without changing the model.

The trap is overfitting to one happy prompt. If the task only passes when you phrase it perfectly, you measured prompt luck. A good custom benchmark has a fixture, a known expected answer, and enough variation to catch brittle behavior.

Try a tiny Cursor review eval

You can borrow the Oqoqo lesson without building a full benchmark harness on day one. Pick one review task that already causes friction in Cursor. Keep it small enough that a human can verify it in ten minutes.

A good first task is a cursor ai code review and refactoring example from a real pull request. For instance: review a change that replaces duplicated date formatting in packages/billing, confirm no invoice totals changed, and suggest only safe refactors. That gives the agent a realistic job with a clear evidence trail.

Use Cursor rules to make the boundary visible. Put repo-wide conventions in AGENTS.md, and put local review constraints in a scoped .mdc rule. If you are experimenting with the related training topic, this is also where a Cursor custom subagent or Cursor skill earns its keep: it should make the eval easier to pass for the right reason.

Here is a small rule stub you can adapt:

---
description: Use during review evals for billing changes in packages/billing
globs:
  - packages/billing/**
alwaysApply: false
---
When reviewing billing changes:
- Check tests or fixtures before suggesting refactors.
- Do not approve behavior changes without invoice evidence.
- Mention the exact file and line for each blocking issue.
- Separate correctness findings from style suggestions.

The trap is letting the agent grade itself. Save the transcript, diff, command output, and final review. Then have a human compare the answer against a short answer key. Agents are very good at sounding done before the evidence is done.

Copy this one-hour mini-eval plan

Use this as a tiny internal version of the Oqoqo idea. It is not a replacement for a benchmark platform. It is a safe way to learn what your current Cursor workflow can and cannot measure.

Task fixture

• Choose one merged pull request with a known review issue.
• Reset a branch to the pre-review state.
• Keep the diff under 300 lines if possible.
• Add one scoped Cursor rule or AGENTS.md boundary that should affect the review.

Agent instruction

• Ask Cursor Agent to review the diff, not rewrite it first.
• Tell it to cite files, tests, and commands used.
• Allow read-only inspection commands.
• Disallow pushing, deleting files, or changing fixtures.

Evidence checklist

• Did the agent read the scoped rule or repo boundary?
• Did it inspect the changed files rather than summarize the PR title?
• Did it run or request the smallest relevant test?
• Did it find the known issue?
• Did it avoid inventing failures that are not in the diff?
• Did it separate blocking findings from refactor suggestions?

Pass condition

• Pass: finds the known issue, cites evidence, and avoids unsafe edits.
• Partial: finds the issue but lacks evidence or mixes review with broad refactoring.
• Fail: misses the issue, changes code without permission, or fabricates a test result.

Run the same task twice: once with only the raw prompt, and once with the Cursor rule enabled. If the rule improves evidence quality without making the agent noisy, you learned something concrete. If nothing changes, your rule description may not be activating clearly enough.

For a broader example of why shared context matters across coding agents, see Vibsync Shares Memory Across Cursor, Claude, Codex. Memory and evals are different problems, but they meet at the same question: did the agent use the right context at the right time?

Common questions

• Does Oqoqo replace cursor code review?

  No, Oqoqo does not replace code review; it helps measure whether an agent-facing review workflow behaves consistently. A useful eval can check one known review issue, one scoped rule, and one evidence standard, but a human still owns judgment on risk, maintainability, and product intent.

• Can I use this idea without a public benchmark?

  Yes, you can start with one private fixture and a written answer key. The important artifact is not a leaderboard; it is a rerunnable task with the same repo state, same permissions, same expected evidence, and a pass/fail result you can compare after changes.

• What should a Cursor eval measure first?

  Measure whether Cursor Agent follows your local repo boundary before measuring broad coding ability. One AGENTS.md note, one .mdc rule, and one known review defect are enough to reveal whether the workflow is using context or merely producing plausible review prose.

• Is a cursor ai code review and refactoring example enough evidence?

  One example is enough to debug the workflow, but not enough to claim reliability. Treat the first run as a calibration pass, then add two near-neighbor tasks: one with a real bug, one with only style cleanup, and one where refactoring would be unsafe.

Best ways to use this research

• Best for: deciding whether your agent-facing interfaces are clear enough for real work, especially Cursor rules, MCP tools, CLIs, SDKs, and review flows.
• Best first artifact: a one-page eval receipt with the fixture branch, prompt, enabled rules, commands run, final review, and human pass/fail note.
• Best comparison angle: compare the same task with and without a scoped Cursor rule before comparing models. If the rule changes the result, your interface design mattered.
• Best limitation to remember: realistic evals are more honest than sterile tasks, but they are also easier to contaminate. Keep fixtures frozen and expected answers written before the run.

Further reading

• Oqoqo — source
• Cursor — Agent
• Cursor — Rules

Next step

Pick one real review your team already understands, turn it into the mini-eval above, and run it once in Cursor. The first useful result is not a score; it is the first place your agent lacked evidence.

One methodology lens

One useful way to read this through our methodology is the Plan step: delegate first-pass decomposition and dependency mapping, review the sequencing and assumptions, and keep ownership of scope and priorities. If that split is still fuzzy, the workflow usually is too.