Claude Code vs GitHub Copilot: Measured on Real Tasks
Most "Claude Code vs GitHub Copilot" pages compare two different things: Copilot's autocomplete in the editor against Claude Code's agent in the terminal, and conclude with "use both". That was fair in 2025. In 2026, both have a terminal agent with a headless mode, so we compared what can actually be compared: Claude Code against GitHub Copilot CLI, on four real tasks with objective checks, two runs each, headless, with the commands, prompts and raw data. Measured on 2026-11-03 with Claude Code 2.1.317 (default model, claude-opus-5-5) and Copilot CLI 1.0.105 (default model on our Copilot Pro account, claude-haiku-4.5), plus a third configuration that runs Claude Code on Haiku 4.5 to separate the model from the CLI.
TL;DR
- Quality: Claude Code wins, because of the model. Claude Code passed 8 tasks out of 8 and found all 3 planted bugs in both reviews. Copilot CLI on its default Claude Haiku 4.5 passed 5 out of 8 and found 1 planted bug per review. Claude Code on the same Haiku 4.5 also passed 5 out of 8.
- Cost: close per run, Claude Code ahead per passed task. 15.9 AI credits ($0.159) per Copilot run against $0.187 per Claude Code run at API prices; per passed task, $0.255 against $0.187. On a legacy annual Copilot plan, each run cost 0.33 premium request, about 1.3 cents.
- Speed: Claude Code, 2.4x faster. 366 seconds for the eight runs against 870, with 40 tool calls against 216.
Claude Code vs GitHub Copilot at a glance
| Claude Code | GitHub Copilot | |
|---|---|---|
| Maker | Anthropic | GitHub (Microsoft) |
| Surfaces | Terminal, VS Code and JetBrains extensions, desktop app, web | Inline completions in VS Code, Visual Studio, JetBrains, Xcode and other editors, Copilot Chat, Copilot CLI, cloud coding agent, code review on pull requests |
| Models | Claude only | Anthropic, OpenAI, Google, xAI, Microsoft, Moonshot (the CLI lists 31 model ids, including 11 Claude models) |
| Default model (measured) | claude-opus-5-5 | claude-haiku-4.5 on our account (Pro, annual, legacy billing) |
| Headless command | claude -p (docs) |
copilot -p (docs) |
| Machine-readable output | --output-format json or stream-json, with total_cost_usd |
--output-format json (JSONL events), --usage-output-file for tokens per model |
| Skip every permission prompt | --dangerously-skip-permissions |
--allow-all (alias --yolo) |
| Instruction files | CLAUDE.md |
.github/copilot-instructions.md, AGENTS.md, CLAUDE.md and GEMINI.md, combined |
| MCP, skills, hooks, custom agents | Yes | Yes, plus a built-in GitHub MCP server |
| Entry plan | Claude Pro, $20/month ($17 billed annually) | Copilot Pro, $10/month with $15 of AI credits |
| Pay per token | API key, Opus 5.5 at $4 / $20 per million input / output tokens | AI credits beyond the included amount, at each model's API rate |
| License | Proprietary | Proprietary |
Prices from claude.com/pricing, GitHub Copilot plans and GitHub's models and pricing on 2026-11-03. The instruction files are what copilot instruction list printed in a test folder holding all four.
They are no longer different categories
Copilot in 2026: completions, CLI, cloud agent
Copilot started as autocomplete, and that part is still there: inline completions and next edit suggestions, which consume no AI credits on paid plans. Around it, GitHub has built Copilot Chat in the editors, a cloud coding agent you assign an issue to, code review on pull requests, and Copilot CLI, a terminal agent with a headless -p mode, a plan mode, an autopilot mode, a --fleet mode that runs subagents in parallel, and a --max-ai-credits cap per session. One subscription gives access to models from several vendors, Claude among them.
Claude Code in 2026: terminal agent, IDE, web
Claude Code is Anthropic's agent, and only an agent: you give it a task, it reads, edits, runs commands and tests. It runs in the terminal, in VS Code and JetBrains through extensions, in a desktop app and on the web, always with Claude models, on a Claude subscription or an API key. It has no inline completion.
What only one of them does
- Only Copilot: inline completion in many editors, models from several vendors behind one subscription, a cloud agent started from a GitHub issue, code review on github.com, and third-party agents: GitHub runs Anthropic's Claude agent and OpenAI's Codex on github.com, GitHub Mobile and VS Code, in public preview since February 2026, listed on Pro+ and Max on the plans page of 2026-11-03.
- Only Claude Code: Anthropic's newest models, Opus included, on the $20 entry plan (Copilot's plans page sells "premium models including Opus" as a Pro+ feature, and our annual Pro plan offers only Opus 4.8, at 27 premium requests per prompt), and permission modes such as auto mode, where a classifier reviews each action.
Both now have MCP, skills, hooks and custom agents or subagents, so that is no longer a differentiator.
How we tested (and how you can rerun it)
Same protocol as our Codex vs Claude Code and Gemini CLI vs Claude Code benchmarks: same repository, same four tasks, same prompts, same check scripts. Claude Code moved from 2.1.296 to 2.1.317 since the last run, so we reran it rather than reuse old numbers. The raw data archive (2.6 MB) has the prompts, the hidden tests, the scripts, the Dockerfile, the full transcript and Copilot session events of each run, the computed metrics and our scoring of the reviews.
The repository and the four tasks
The repository is python-humanize/humanize (MIT, 1,725 lines of Python in src/, about 860 tests in 7 seconds), pinned at commit 785e5dcc0d0308ad0dff3f6cc0faa7085ad0375b.
| Task | Starting commit | Objective check |
|---|---|---|
| Bug fix from a real issue | f971127, the parent of the fix for PR #346 |
The upstream regression test, copied in after the run, plus the full suite |
| Feature from a spec | 785e5dc |
16 hidden acceptance tests for two new natural_list parameters, copied in after the run, plus the full suite |
| Refactor, no behavior change | 785e5dc |
Move scientific() and metric() to a new module: tests/ untouched, old and new import paths return the same objects, docs updated, doctests and suite green, no duplicated helper |
| Code review | a0f0aa3: 785e5dc plus a parse_size() function with 3 planted bugs |
Planted bugs found, false alarms, and no file modified |
The three planted bugs are a missing + 1 in the binary units branch (parse_size("1.0 KiB") returns 1), an int() truncation of a float product (parse_size("8.2 MB") returns 8199999) and a regular expression with a nested quantifier, (?:\d+)+, that backtracks exponentially on input like "1" * 24 + "!". The Codex post shows the full diff and prompts.
One check is new. The refactor prompt says "Helpers still needed by number.py must not be duplicated", and the scripted checks did not test it, because no earlier run had done it. Two runs this time copied the _format_not_finite helper into the new module instead of importing it. We added the check after the runs (scripts/postcheck.sh in the archive) and count those two runs as failed.
The exact commands and prompts
Every run went into its own disposable Docker container (docker run --rm), with a fresh git worktree of the starting commit and its own virtualenv. Only the CLI binary and a read-only copy of that CLI's sign-in file were mounted (for Copilot, a config.json reduced to its sign-in fields): no home directory, no other credentials.
docker run --rm -e HOME=/home/bench \
-v "$PWD/repo:/repo:ro" -v "$PWD/tasks:/bench/tasks:ro" \
-v "$PWD/scripts:/bench/scripts:ro" -v "$PWD/results:/out" \
-v "$(command -v copilot):/usr/local/bin/copilot:ro" \
-v "$PWD/creds/copilot-config.json:/creds/copilot-config.json:ro" \
cbench:1 /bench/scripts/in-container.sh copilot bugfix 1Inside the container, same prompt text, permissionless:
# Claude Code 2.1.317, default model (add --model claude-haiku-4-5 for the same-model runs)
env -i HOME="$HOME" PATH="$PATH" IS_SANDBOX=1 \
claude -p "$PROMPT" --output-format stream-json --verbose \
--dangerously-skip-permissions \
--setting-sources project,local --strict-mcp-config --no-session-persistence < /dev/null
# GitHub Copilot CLI 1.0.105, default model
env -i HOME="$HOME" PATH="$PATH" COPILOT_AUTO_UPDATE=false \
copilot -p "$PROMPT" --allow-all --no-ask-user --output-format json \
--usage-output-file "$OUT.usage.json" < /dev/null--allow-allis--allow-all-tools --allow-all-paths --allow-all-urls;--no-ask-userremoves the tool that lets the agent stop and ask a question.COPILOT_AUTO_UPDATE=falsekeeps the CLI from updating itself in the middle of a benchmark, as GitHub recommends for CI.- Copilot ran with its defaults, including the built-in GitHub MCP server (18 tools in every run). Claude Code ran without MCP servers (
--strict-mcp-config), as in our previous benchmarks. - The containers start with an empty home directory, so no personal settings or instruction files reach any run.
IS_SANDBOX=1is there because the containers run as root, and Claude Code refuses--dangerously-skip-permissionsas root otherwise.
Every prompt but the review ends with: "The project virtualenv is ready in .venv: run the tests with .venv/bin/python -m pytest -q. Work only inside this repository. Do not commit." The review prompt ends with "Do not modify any file."
What we measured, and how we turn AI credits into dollars
- Pass or fail on the checks above, run by a script after each run; reviews scored by hand, a planted bug counting only if the finding names the defect and a correct consequence (
review-scores.jsonin the archive). - Wall time from launch to exit, 24 runs four containers at a time on a 4 vCPU machine.
- Tokens as each CLI reports them: Claude Code's final
resultevent, and for Copilot thesession.usage_recordlines of~/.copilot/session-state/<id>/events.jsonl, one per model call (input, cache reads, cache writes, output, reasoning). Copilot's input count includes cache reads and writes, so we subtract them to get uncached input. - Cost on the same scale for both. Since June 2026, GitHub bills Copilot in AI credits "at the published API rate for each model", 1 credit = $0.01. Claude Haiku 4.5 costs $1 per million input tokens, $0.10 cached, $1.25 cache writes and $5 output, the same as on Anthropic's API. Claude Code's cost is its own
costUSDat Anthropic list prices (Opus 5.5: $4 input, $8 for 1-hour cache writes, $0.20 cache reads, $20 output), which is what an API key pays. - Premium requests, because our Copilot account is an annual Pro plan still on the old billing: each run was charged
premiumRequests: 0.33, the Haiku 4.5 multiplier, once per prompt.
Results
| Task | Passed (Claude Code / Copilot) | Wall time, median | Cost per run, median | Copilot cost vs Claude Code |
|---|---|---|---|---|
| Bug fix | 2/2 / 2/2 | 34.4 s / 164.9 s | $0.126 / $0.256 | 2.0x more |
| Feature | 2/2 / 2/2 | 34.6 s / 58.1 s | $0.123 / $0.068 | 1.8x less |
| Refactor | 2/2 / 1/2 | 77.4 s / 157.2 s | $0.382 / $0.250 | 1.5x less |
| Code review (3 planted bugs) | 3/3 twice / 1/3 twice | 36.8 s / 54.6 s | $0.117 / $0.062 | 1.9x less |
Bug fix from a real issue
| Claude Code (run 1, run 2) | Copilot CLI (run 1, run 2) | |
|---|---|---|
| Hidden test and suite | pass, pass | pass, pass |
| Wall time | 34.0 s, 34.8 s | 79.7 s, 250.2 s |
| Input tokens (of which from cache) | 101.6k (90.3k), 101.5k (97.0k) | 586k (557k), 2.17M (2.12M) |
| Tokens written to cache | 11.2k, 4.5k | 28.5k, 51.3k |
| Output tokens | 2,387, 1,996 | 5,408, 23,539 |
| Tool calls | 4, 4 | 22, 51 |
| Cost | $0.156, $0.095 | $0.119 (11.8 credits), $0.394 (39.4 credits) |
| Lines changed | +10 / -1, +11 / -1 | +7 / -1, +11 / -1 |
All four runs fixed the bug. Claude Code wrote the same integer comparison both times, rounded_value == powers[ordinal + 1] // power, which is the upstream fix. Copilot found two other correct formulations: a Decimal comparison in run 1, and in run 2 a test on rounded_value >= 1000.0. Run 2 got there slowly: 51 tool calls, 9 of them edits, about twenty Python experiments on float precision, and two git stash round trips to compare its output with the original code. That one run cost more than both Claude Code runs together.
Feature from a spec
| Claude Code (run 1, run 2) | Copilot CLI (run 1, run 2) | |
|---|---|---|
| Hidden tests and suite | pass, pass | pass, pass |
| Wall time | 31.3 s, 37.9 s | 59.4 s, 56.9 s |
| Input tokens (of which from cache) | 78.9k (69.0k), 80.0k (76.4k) | 326k (309k), 219k (203k) |
| Tokens written to cache | 9.9k, 3.6k | 17.3k, 16.1k |
| Output tokens | 2,626, 2,824 | 4,554, 4,112 |
| Tool calls | 3, 3 | 16, 11 |
| Cost | $0.146, $0.101 | $0.075 (7.5 credits), $0.061 (6.1 credits) |
| Lines changed | +90 / -7, +88 / -7 | +64 / -5, +66 / -5 |
A precise spec suits a small model: Copilot passed the 16 hidden tests twice, for half the price of Claude Code. Claude Code needed three tool calls and wrote a few more tests; Copilot needed 11 to 16, all view, bash and two edit calls.
Refactor without behavior change
| Claude Code (run 1, run 2) | Copilot CLI (run 1, run 2) | |
|---|---|---|
| Scripted checks (tests untouched, API, docs, doctests, suite) | pass, pass | pass, pass |
| Helper not duplicated | yes, yes | yes, no |
| Wall time | 89.5 s, 65.3 s | 160.4 s, 154.1 s |
| Input tokens (of which from cache) | 408k (383k), 228k (207k) | 1.65M (1.61M), 1.30M (1.26M) |
| Tokens written to cache | 24.6k, 21.7k | 42.6k, 37.8k |
| Output tokens | 7,877, 5,887 | 10,823, 11,408 |
| Tool calls | 12, 7 | 51, 40 |
| Cost | $0.431, $0.332 | $0.269 (26.9 credits), $0.230 (23.0 credits) |
| Lines changed | +153 / -132, +158 / -127 | +161 / -129, +171 / -131 |
The trap is a helper used by both modules, _format_not_finite, which turns a naive move into a circular import. Claude Code solved it twice, once by importing the helper inside the moved functions, once with a lazy module-level __getattr__ in number.py. Copilot's first run imported the helper from number.py and re-exported the two functions. Its second run avoided the cycle by pasting a copy of the helper into notation.py, exactly what the prompt ruled out. Every scripted check passed, so only reading the diff catches it.
Code review with planted bugs
| Claude Code (run 1, run 2) | Copilot CLI (run 1, run 2) | |
|---|---|---|
| Planted bugs found | 3/3, 3/3 | 1/3, 1/3 |
| False alarms | 0, 0 | 0, 0 |
| Extra valid findings | 3, 3 | 2, 2 |
| Repository left untouched | yes, yes | yes, yes |
| Wall time | 32.2 s, 41.3 s | 50.6 s, 58.6 s |
| Input tokens (of which from cache) | 79.4k (70.9k), 102.1k (97.5k) | 207k (192k), 254k (237k) |
| Output tokens | 2,206, 2,621 | 3,896, 4,581 |
| Tool calls | 3, 4 | 10, 15 |
| Cost | $0.126, $0.109 | $0.058 (5.8 credits), $0.067 (6.7 credits) |
This is where the two setups split. Both Claude Code reviews found the missing + 1, the truncation (parse_size("4.1 MB") returns 4099999) and the ReDoS, which it timed at 2.3 seconds for 24 digits. Both Copilot reviews found the binary bug and the GNU suffixes that parse_size() rejects, then missed the other two: the first never mentioned the regular expression, the second called (?:\d+)+ "redundant" and "functionally equivalent" to \d+, which is true for what it matches and wrong for how long it takes. Neither connected int() to wrong results on ordinary sizes.
A reviewer that misses a denial of service on untrusted input is cheap for the wrong reason. If you run reviews every day, as our daily pull request review template does, pick the model before the CLI: see below.
Totals: tokens, time and cost per passed task
| Over 8 runs each | Claude Code (Opus 5.5) | Copilot CLI (Haiku 4.5) |
|---|---|---|
| Passed | 8/8 | 5/8 |
| Total wall time | 366 s | 870 s |
| Input tokens | 1.18M (92% read from cache) | 6.71M (97% from cache) |
| Uncached input tokens | 96 | 1,174 |
| Tokens written to cache | 88.6k (1-hour cache) | 225k (5-minute cache) |
| Output tokens | 28.4k | 68.3k |
| Model calls / tool calls | 48 / 40 | 215 / 216 |
| Cost | $1.50 | $1.27 (127 AI credits) |
| Cost per run | $0.187 | $0.159 |
| Cost per passed task | $0.187 | $0.255 |
| Premium requests (legacy annual plan) | – | 2.64 (0.33 per run) |
Haiku 4.5 input and output tokens cost four times less than Opus 5.5's, yet a Copilot run cost only 15% less than a Claude Code run. Copilot read 5.7 times more input and wrote 2.4 times more output, because it worked in many small steps: 215 model calls against 48. Where the money went:
- Claude Code: 47% cache writes ($0.71), 38% output ($0.57), 15% cache reads ($0.22). Same profile as in our previous benchmarks: almost nothing uncached, and a 1-hour cache that one-shot runs rarely reuse.
- Copilot CLI: 51% cache reads ($0.65), 27% output ($0.34), 22% cache writes ($0.28). Each of its 215 calls resends the conversation, and even at $0.10 per million those reads add up.
Same model, different CLI: Copilot CLI running Claude
Copilot's default model on our account is a Claude model, so the cleanest way to separate the model from the agent loop was to run Claude Code on it too, claude -p --model claude-haiku-4-5, same tasks, two runs each:
| Over 8 runs each, Claude Haiku 4.5 in both | Claude Code | Copilot CLI |
|---|---|---|
| Passed | 5/8 | 5/8 |
| Failed | both reviews (1/3 each), one refactor (duplicated helper) | both reviews (1/3 each), one refactor (duplicated helper) |
| Total wall time | 696 s | 870 s |
| Model calls / tool calls | 131 / 123 | 215 / 216 |
| Input / output tokens | 5.16M / 81.3k | 6.71M / 68.3k |
| Cache writes | 218k at $2 per million (1 hour) | 225k at $1.25 per million (5 minutes) |
| Cost | $1.34 | $1.27 |
The results are almost symmetrical. With the same model, Claude Code failed the same tasks in the same way: its two reviews found only the binary bug, and one of them wrote that the nested quantifier "doesn't cause catastrophic backtracking (due to anchors)", which is false; one refactor also duplicated the helper. So on these tasks, quality follows the model, not the CLI. The CLI changes the path: Claude Code made 43% fewer tool calls and finished 20% sooner, and the two ended up at about the same cost, because Claude Code's 1-hour cache writes cost 60% more per token than Copilot's 5-minute ones.
Two more data points from the same runs:
- Opus 5.5 against Haiku 4.5 inside Claude Code: 8/8 against 5/8, 1.9 times faster, and only 12% more expensive in total ($1.50 against $1.34), because Opus needed 48 model calls where Haiku needed 131.
- Copilot CLI with a stronger model: one side review with
--model claude-opus-4.8, the strongest Claude model our plan offers. It found the ReDoS and measured it (about 8.8 seconds for 26 digits), missed the truncation, and added three valid findings: 2 planted bugs out of 3, in 125 seconds, for 41.1 AI credits ($0.41), or 27 premium requests on our legacy plan ($1.08 at the $0.04 overage price). That is 3.5 times the median cost of a Claude Code review, with a weaker result.
If you use Copilot, the model picker is the main quality setting. The current plans page lists Opus 5.5 for monthly plans; we could not run it, since annual plans on legacy billing don't get new models.
Pricing in practice after the switch to AI credits
Copilot plans: Free, Pro $10, Pro+ $39, Max $100, Business and Enterprise
Since June 1, 2026, Copilot usage consumes GitHub AI Credits, priced from tokens (input, output, cached) at each model's API rate. Plans on 2026-11-03:
| Plan | Price | AI credits included each month |
|---|---|---|
| Free | $0 | Limited chat and agent use; 2,000 completions; Copilot CLI included |
| Pro | $10 | $10 base + $5 flex = 1,500 credits |
| Pro+ | $39 | $39 base + $31 flex = 7,000 credits |
| Max | $100 | $100 base + $100 flex = 20,000 credits |
| Business | $19 per user | $19 |
| Enterprise | $39 per user | $39 |
What else to know:
- Completions and next edit suggestions consume no credits on paid plans. Copilot code review consumes credits and GitHub Actions minutes.
- Annual Pro and Pro+ subscribers stay on premium requests until renewal, then move to Free unless they switch to a monthly plan. On that billing, each prompt to Copilot CLI costs one premium request times the model multiplier (0.33 for Haiku 4.5, 27 for Opus 4.8), and tool calls are not counted (premium requests, multipliers). That is why a 51-call run cost us the same 0.33 request as an 11-call run.
- Training data: since April 24, 2026, GitHub uses interaction data from Free, Pro and Pro+ to train models unless you opt out in your Copilot settings; Business and Enterprise are excluded (GitHub blog).
Claude plans: Pro $20, Max $100, Ultra $200, Team seats, API
| Plan | Price | Claude Code |
|---|---|---|
| Free | $0 | Not included |
| Pro | $20/month, $17/month billed annually | Included |
| Max | $100/month | Included |
| Ultra | $200/month | Included |
| Team | Standard seat $20/month annual ($25 monthly), premium seat $100 ($125) | Included |
| Enterprise | $20 per seat per month billed annually (20 seats minimum), plus usage at API rates | Included |
| API | Opus 5.5: $4 / $20 per million input / output tokens | Pay per token |
Source: claude.com/pricing on 2026-11-03. Subscriptions have usage limits over a five-hour window and a week; Anthropic publishes no token count for them.
What one month of real use costs on each
Projected from our measured averages, at the prices of 2026-11-03:
- 100 tasks like ours: about $18.70 with Claude Code on an API key, about 1,590 AI credits ($15.90) with Copilot CLI on Haiku 4.5. Copilot Pro's 1,500 included credits cover about 94 of them.
- An unattended job with 20 tasks every weekday (about 440 a month, like a nightly triage or a weekly dependency update at scale): about $82 with Claude Code at API prices; about 7,000 credits with Copilot, which is what Pro+ includes for $39.
- On a legacy annual Copilot Pro plan: 440 runs at 0.33 premium request is 145 requests, about half of the 300 included each month. Until renewal, nothing on this page is cheaper.
Two warnings with those numbers. Copilot's cheaper run came with 3 failures out of 8: per passed task, it is the more expensive of the two. And on a Claude subscription, the same Claude Code runs cost you limits instead of dollars: none of our runs hit a limit, but nobody publishes how many tasks a five-hour window holds.
Running them unattended: headless mode and permissions
claude -p vs copilot -p: flags and output formats
| Claude Code | Copilot CLI | |
|---|---|---|
| One prompt, no TTY | claude -p "..." |
copilot -p "..." |
| Output | --output-format json (one object) or stream-json (events) |
--output-format json (JSONL events), -s for the answer only |
| Usage in the output | usage, modelUsage, total_cost_usd in the final result |
premiumRequests and session duration in the final result; tokens per model with --usage-output-file, and per call in ~/.copilot/session-state/<id>/events.jsonl |
| Model | --model |
--model (or COPILOT_MODEL) |
| Spending cap | – | --max-ai-credits |
| Resume a session | --resume <id>, --session-id <uuid> |
--resume <id>, --session-id <uuid> |
| Authentication in CI | ANTHROPIC_API_KEY or a subscription login |
COPILOT_GITHUB_TOKEN, GH_TOKEN or GITHUB_TOKEN |
| Pin the version | Install a fixed version | COPILOT_AUTO_UPDATE=false |
Permissions: --dangerously-skip-permissions vs --allow-all, and the silent refusal without them
We ran each CLI headless without its skip flag, in a container, with a prompt that needs a shell command (uname -s > os.txt):
| Exit code | File created | Where the refusal shows | |
|---|---|---|---|
claude -p (no skip flag) |
0 | no | permission_denials in the JSON (two attempts, Bash then Write) |
copilot -p (no --allow-all) |
0 | no | tool.execution_complete events with "Permission denied because no interactive user response was available"; 0.33 premium request still charged |
Both exit with code 0, and both answers say politely that they could not do it. A CI job that only checks the exit code reports success for a run that did nothing. Check the denial fields, or grant exactly what the job needs: Claude Code's permission modes and allow rules, Copilot's --allow-tool, --deny-tool (which wins over --allow-all-tools) and --assisted-approval, where a model judges each request. Keep --dangerously-skip-permissions and --allow-all (alias --yolo) for disposable containers or VMs, as we did. Our post on Claude Code's --dangerously-skip-permissions tests what that flag still blocks, and lists the equivalent flag in each CLI.
Gotchas we hit
- An MCP server that never answers blocks
copilot -pforever. On our VPS on 2026-10-12, an MCP server listed in~/.copilot/mcp-config.jsonstopped responding, and every headless Copilot run sat until its timeout with almost no CPU. The clue is in~/.copilot/logs/process-*.log("Failed to list MCP tools"). Test withcopilot --disable-mcp-server <name> -p "pong", and in CI pass--disable-builtin-mcpsor only the servers you need. - The built-in GitHub MCP server is on by default. It loaded 18 tools in every run; in our smoke test, the 39 tool definitions took about 6,500 tokens per request. It is useful if your task touches issues and pull requests, and dead weight otherwise.
- Folder trust. Copilot asks to trust a folder in interactive mode. In prompt mode,
--trust-folderis what lets it load the repository's hooks, MCP servers and plugins for one session (settingCOPILOT_ALLOW_ALLto exactlytruealso trusts the folder); GitHub's advice is to use it only on configuration you have reviewed. None of our headless runs was blocked. Claude Code never asks in-pmode. - Instruction files. Copilot CLI reads
CLAUDE.md,AGENTS.md,GEMINI.mdand.github/copilot-instructions.mdtogether. OneCLAUDE.mdserves both CLIs;--no-custom-instructionsturns them off for Copilot.
Which one should you use?
When Claude Code is the better pick
- Reviews. It was the only setup that found all three planted bugs, twice, with three extra valid findings per review, at $0.12 per review.
- Tasks with a trap, like the circular import in the refactor, and anything where a human is waiting: 2.4 times faster in total, 4.8 times faster on the median bug fix.
- Opus on a $20 plan. Claude Pro includes it; on Copilot, Opus is sold as a Pro+ feature.
When GitHub Copilot is the better pick
- You want inline completion. Claude Code has none, and Copilot's consumes no credits.
- Well-specified, small changes billed per token. On the feature task, Copilot on Haiku 4.5 passed twice at half Claude Code's price.
- You are on an annual Copilot plan, where a whole agent run costs 0.33 premium request: nothing is cheaper until renewal.
- Your team lives in GitHub: issues assigned to the cloud agent, code review on pull requests, Business or Enterprise seats with data kept out of training, and one bill for several model vendors.
Claude Code vs Copilot vs Codex vs Cursor: the short view
| Claude Code | Codex CLI | Copilot CLI | |
|---|---|---|---|
| Measured on | 2026-10-06, 2026-10-13, 2026-11-03 | 2026-10-06 | 2026-11-03 |
| Default model | claude-opus-5-5 | gpt-6.1-sol | claude-haiku-4.5 (our account) |
| Passed | 8/8 each time | 8/8 | 5/8 |
| Cost per passed task | $0.181, $0.169, $0.187 | $0.074 | $0.255 |
| Total wall time, 8 runs | 452 s, 318 s, 366 s | 389 s | 870 s |
Same tasks and checks on all three dates, with different CLI versions and parallelism, so compare orders of magnitude, not decimals. Codex remains the cheapest correct option on these tasks, Claude Code the fastest with the deepest reviews, and Copilot CLI depends on the model you give it. The details are in Codex vs Claude Code and Gemini CLI vs Claude Code. Cursor is an editor with its own agent rather than a CLI you script, and is not part of this benchmark.
Using Claude Code and Copilot together
The results suggest a split by task type rather than a winner: Copilot on a cheap model did well when the spec left no room for interpretation, and only Claude Code on Opus 5.5 caught every planted bug. So let Copilot implement well-specified changes, and let Claude Code review them before anything is pushed.
We ran this benchmark with plain scripts, but in SideHub the same split is a workflow: a list of steps run by an agent on your own machine, where each step can set its own CLI with provider (claude, codex, agy, copilot or gemini):
format: sidehub.workflow/v1
name: Copilot implements, Claude Code reviews
defaultProvider: claude
parameters:
- { name: issue, label: Issue number, type: string, required: true }
steps:
- name: Implement the issue
provider: copilot
prompt: |
Read GitHub issue #{{inputs.issue}} with `gh issue view`, implement it on a new branch
and run the test suite. Do not push.
- name: Review the diff
inputsFrom: ["1"]
prompt: |
Review the uncommitted diff of this repository for correctness, edge cases, security and
performance. List each problem with file, line, severity and a reproducing input.
Do not modify any file.Each step runs as its own run, with its CLI, duration, exit code and token usage; for Copilot, SideHub reads the same ~/.copilot/session-state/<id>/events.jsonl file we used here, and shows the cost as an API equivalent. That is how to rerun this comparison on your own repository: same prompt, one step per CLI, and compare the runs. Three honest limits, all from our own use: under a per-tool approval policy, SideHub checks Claude Code's tool calls one by one but cannot do that for Copilot, so it launches Copilot without --allow-all and tools that need approval are refused; plan-limit detection relies on a Claude Code hook, while Copilot gives no structured signal; and the dead MCP server above is a Copilot problem SideHub cannot fix for you.
The template gallery has workflows to start from: Claude codes, Codex reviews is the same cross-CLI pattern with Codex, and nightly issue triage and dependency updates are unattended jobs where the cost per run adds up.
Limitations of this benchmark
- Small sample. Two runs per task per configuration, one run for the Opus 4.8 side review. One Copilot bug fix took 250 seconds against 80 for its twin, which shows how much a single run moves a median.
- Copilot's default model is our account's default. Our Copilot Pro plan is annual and still on premium requests; it ran Claude Haiku 4.5 by default and could not select Opus 5.5 or Sonnet 5.5. A monthly plan on AI credits may default to another model and offer newer ones; the cost per task at credit prices would change with it.
- One repository, small tasks. A tidy Python library with fast tests, and tasks of 30 seconds to 4 minutes.
- Our planted bugs, our scoring. We wrote the review diff and scored the reviews by hand; the scores and the reviews are in the archive.
- A check added after the runs. The duplicated-helper check came from reading the diffs. It changed one Copilot run and one Claude Code on Haiku run from pass to fail, and no Claude Code on Opus run.
- Possible contamination. The bug fix comes from a public pull request of 2026-09-16, and both Claude Code runs on Opus wrote the upstream comparison.
- Dated prices. List prices of 2026-11-03. GitHub's per-model rates and Anthropic's prices both move with new models.
The verdict
On four everyday tasks, Claude Code with Opus 5.5 passed everything and wrote the only complete reviews; Copilot CLI with its default Claude Haiku 4.5 passed 5 out of 8, for about the same price per run and 2.4 times the time. Running Claude Code on Haiku 4.5 gave the same 5 out of 8, so the comparison that matters is less "Claude Code vs GitHub Copilot" than "which model is behind it". Pick Copilot for inline completion, GitHub-native work, an annual plan that makes agent runs almost free, or well-specified changes on a cheap model; pick Claude Code for reviews, tricky changes and Opus on a $20 plan. Then rerun these four tasks on your own repository: the commands, prompts and data are all above.