Claude Code vs GitHub Copilot: Measured on Real Tasks

Most "Claude Code vs GitHub Copilot" pages compare two different things: Copilot's autocomplete in the editor against Claude Code's agent in the terminal, and conclude with "use both". That was fair in 2025. In 2026, both have a terminal agent with a headless mode, so we compared what can actually be compared: Claude Code against GitHub Copilot CLI, on four real tasks with objective checks, two runs each, headless, with the commands, prompts and raw data. Measured on 2026-11-03 with Claude Code 2.1.317 (default model, claude-opus-5-5) and Copilot CLI 1.0.105 (default model on our Copilot Pro account, claude-haiku-4.5), plus a third configuration that runs Claude Code on Haiku 4.5 to separate the model from the CLI.

TL;DR

Claude Code vs GitHub Copilot at a glance

Claude Code GitHub Copilot
Maker Anthropic GitHub (Microsoft)
Surfaces Terminal, VS Code and JetBrains extensions, desktop app, web Inline completions in VS Code, Visual Studio, JetBrains, Xcode and other editors, Copilot Chat, Copilot CLI, cloud coding agent, code review on pull requests
Models Claude only Anthropic, OpenAI, Google, xAI, Microsoft, Moonshot (the CLI lists 31 model ids, including 11 Claude models)
Default model (measured) claude-opus-5-5 claude-haiku-4.5 on our account (Pro, annual, legacy billing)
Headless command claude -p (docs) copilot -p (docs)
Machine-readable output --output-format json or stream-json, with total_cost_usd --output-format json (JSONL events), --usage-output-file for tokens per model
Skip every permission prompt --dangerously-skip-permissions --allow-all (alias --yolo)
Instruction files CLAUDE.md .github/copilot-instructions.md, AGENTS.md, CLAUDE.md and GEMINI.md, combined
MCP, skills, hooks, custom agents Yes Yes, plus a built-in GitHub MCP server
Entry plan Claude Pro, $20/month ($17 billed annually) Copilot Pro, $10/month with $15 of AI credits
Pay per token API key, Opus 5.5 at $4 / $20 per million input / output tokens AI credits beyond the included amount, at each model's API rate
License Proprietary Proprietary

Prices from claude.com/pricing, GitHub Copilot plans and GitHub's models and pricing on 2026-11-03. The instruction files are what copilot instruction list printed in a test folder holding all four.

They are no longer different categories

Copilot in 2026: completions, CLI, cloud agent

Copilot started as autocomplete, and that part is still there: inline completions and next edit suggestions, which consume no AI credits on paid plans. Around it, GitHub has built Copilot Chat in the editors, a cloud coding agent you assign an issue to, code review on pull requests, and Copilot CLI, a terminal agent with a headless -p mode, a plan mode, an autopilot mode, a --fleet mode that runs subagents in parallel, and a --max-ai-credits cap per session. One subscription gives access to models from several vendors, Claude among them.

Claude Code in 2026: terminal agent, IDE, web

Claude Code is Anthropic's agent, and only an agent: you give it a task, it reads, edits, runs commands and tests. It runs in the terminal, in VS Code and JetBrains through extensions, in a desktop app and on the web, always with Claude models, on a Claude subscription or an API key. It has no inline completion.

What only one of them does

Both now have MCP, skills, hooks and custom agents or subagents, so that is no longer a differentiator.

How we tested (and how you can rerun it)

Same protocol as our Codex vs Claude Code and Gemini CLI vs Claude Code benchmarks: same repository, same four tasks, same prompts, same check scripts. Claude Code moved from 2.1.296 to 2.1.317 since the last run, so we reran it rather than reuse old numbers. The raw data archive (2.6 MB) has the prompts, the hidden tests, the scripts, the Dockerfile, the full transcript and Copilot session events of each run, the computed metrics and our scoring of the reviews.

The repository and the four tasks

The repository is python-humanize/humanize (MIT, 1,725 lines of Python in src/, about 860 tests in 7 seconds), pinned at commit 785e5dcc0d0308ad0dff3f6cc0faa7085ad0375b.

Task Starting commit Objective check
Bug fix from a real issue f971127, the parent of the fix for PR #346 The upstream regression test, copied in after the run, plus the full suite
Feature from a spec 785e5dc 16 hidden acceptance tests for two new natural_list parameters, copied in after the run, plus the full suite
Refactor, no behavior change 785e5dc Move scientific() and metric() to a new module: tests/ untouched, old and new import paths return the same objects, docs updated, doctests and suite green, no duplicated helper
Code review a0f0aa3: 785e5dc plus a parse_size() function with 3 planted bugs Planted bugs found, false alarms, and no file modified

The three planted bugs are a missing + 1 in the binary units branch (parse_size("1.0 KiB") returns 1), an int() truncation of a float product (parse_size("8.2 MB") returns 8199999) and a regular expression with a nested quantifier, (?:\d+)+, that backtracks exponentially on input like "1" * 24 + "!". The Codex post shows the full diff and prompts.

One check is new. The refactor prompt says "Helpers still needed by number.py must not be duplicated", and the scripted checks did not test it, because no earlier run had done it. Two runs this time copied the _format_not_finite helper into the new module instead of importing it. We added the check after the runs (scripts/postcheck.sh in the archive) and count those two runs as failed.

The exact commands and prompts

Every run went into its own disposable Docker container (docker run --rm), with a fresh git worktree of the starting commit and its own virtualenv. Only the CLI binary and a read-only copy of that CLI's sign-in file were mounted (for Copilot, a config.json reduced to its sign-in fields): no home directory, no other credentials.

docker run --rm -e HOME=/home/bench \
  -v "$PWD/repo:/repo:ro" -v "$PWD/tasks:/bench/tasks:ro" \
  -v "$PWD/scripts:/bench/scripts:ro" -v "$PWD/results:/out" \
  -v "$(command -v copilot):/usr/local/bin/copilot:ro" \
  -v "$PWD/creds/copilot-config.json:/creds/copilot-config.json:ro" \
  cbench:1 /bench/scripts/in-container.sh copilot bugfix 1

Inside the container, same prompt text, permissionless:

# Claude Code 2.1.317, default model (add --model claude-haiku-4-5 for the same-model runs)
env -i HOME="$HOME" PATH="$PATH" IS_SANDBOX=1 \
  claude -p "$PROMPT" --output-format stream-json --verbose \
    --dangerously-skip-permissions \
    --setting-sources project,local --strict-mcp-config --no-session-persistence < /dev/null

# GitHub Copilot CLI 1.0.105, default model
env -i HOME="$HOME" PATH="$PATH" COPILOT_AUTO_UPDATE=false \
  copilot -p "$PROMPT" --allow-all --no-ask-user --output-format json \
    --usage-output-file "$OUT.usage.json" < /dev/null

Every prompt but the review ends with: "The project virtualenv is ready in .venv: run the tests with .venv/bin/python -m pytest -q. Work only inside this repository. Do not commit." The review prompt ends with "Do not modify any file."

What we measured, and how we turn AI credits into dollars

Results

Task Passed (Claude Code / Copilot) Wall time, median Cost per run, median Copilot cost vs Claude Code
Bug fix 2/2 / 2/2 34.4 s / 164.9 s $0.126 / $0.256 2.0x more
Feature 2/2 / 2/2 34.6 s / 58.1 s $0.123 / $0.068 1.8x less
Refactor 2/2 / 1/2 77.4 s / 157.2 s $0.382 / $0.250 1.5x less
Code review (3 planted bugs) 3/3 twice / 1/3 twice 36.8 s / 54.6 s $0.117 / $0.062 1.9x less

Bug fix from a real issue

Claude Code (run 1, run 2) Copilot CLI (run 1, run 2)
Hidden test and suite pass, pass pass, pass
Wall time 34.0 s, 34.8 s 79.7 s, 250.2 s
Input tokens (of which from cache) 101.6k (90.3k), 101.5k (97.0k) 586k (557k), 2.17M (2.12M)
Tokens written to cache 11.2k, 4.5k 28.5k, 51.3k
Output tokens 2,387, 1,996 5,408, 23,539
Tool calls 4, 4 22, 51
Cost $0.156, $0.095 $0.119 (11.8 credits), $0.394 (39.4 credits)
Lines changed +10 / -1, +11 / -1 +7 / -1, +11 / -1

All four runs fixed the bug. Claude Code wrote the same integer comparison both times, rounded_value == powers[ordinal + 1] // power, which is the upstream fix. Copilot found two other correct formulations: a Decimal comparison in run 1, and in run 2 a test on rounded_value >= 1000.0. Run 2 got there slowly: 51 tool calls, 9 of them edits, about twenty Python experiments on float precision, and two git stash round trips to compare its output with the original code. That one run cost more than both Claude Code runs together.

Feature from a spec

Claude Code (run 1, run 2) Copilot CLI (run 1, run 2)
Hidden tests and suite pass, pass pass, pass
Wall time 31.3 s, 37.9 s 59.4 s, 56.9 s
Input tokens (of which from cache) 78.9k (69.0k), 80.0k (76.4k) 326k (309k), 219k (203k)
Tokens written to cache 9.9k, 3.6k 17.3k, 16.1k
Output tokens 2,626, 2,824 4,554, 4,112
Tool calls 3, 3 16, 11
Cost $0.146, $0.101 $0.075 (7.5 credits), $0.061 (6.1 credits)
Lines changed +90 / -7, +88 / -7 +64 / -5, +66 / -5

A precise spec suits a small model: Copilot passed the 16 hidden tests twice, for half the price of Claude Code. Claude Code needed three tool calls and wrote a few more tests; Copilot needed 11 to 16, all view, bash and two edit calls.

Refactor without behavior change

Claude Code (run 1, run 2) Copilot CLI (run 1, run 2)
Scripted checks (tests untouched, API, docs, doctests, suite) pass, pass pass, pass
Helper not duplicated yes, yes yes, no
Wall time 89.5 s, 65.3 s 160.4 s, 154.1 s
Input tokens (of which from cache) 408k (383k), 228k (207k) 1.65M (1.61M), 1.30M (1.26M)
Tokens written to cache 24.6k, 21.7k 42.6k, 37.8k
Output tokens 7,877, 5,887 10,823, 11,408
Tool calls 12, 7 51, 40
Cost $0.431, $0.332 $0.269 (26.9 credits), $0.230 (23.0 credits)
Lines changed +153 / -132, +158 / -127 +161 / -129, +171 / -131

The trap is a helper used by both modules, _format_not_finite, which turns a naive move into a circular import. Claude Code solved it twice, once by importing the helper inside the moved functions, once with a lazy module-level __getattr__ in number.py. Copilot's first run imported the helper from number.py and re-exported the two functions. Its second run avoided the cycle by pasting a copy of the helper into notation.py, exactly what the prompt ruled out. Every scripted check passed, so only reading the diff catches it.

Code review with planted bugs

Claude Code (run 1, run 2) Copilot CLI (run 1, run 2)
Planted bugs found 3/3, 3/3 1/3, 1/3
False alarms 0, 0 0, 0
Extra valid findings 3, 3 2, 2
Repository left untouched yes, yes yes, yes
Wall time 32.2 s, 41.3 s 50.6 s, 58.6 s
Input tokens (of which from cache) 79.4k (70.9k), 102.1k (97.5k) 207k (192k), 254k (237k)
Output tokens 2,206, 2,621 3,896, 4,581
Tool calls 3, 4 10, 15
Cost $0.126, $0.109 $0.058 (5.8 credits), $0.067 (6.7 credits)

This is where the two setups split. Both Claude Code reviews found the missing + 1, the truncation (parse_size("4.1 MB") returns 4099999) and the ReDoS, which it timed at 2.3 seconds for 24 digits. Both Copilot reviews found the binary bug and the GNU suffixes that parse_size() rejects, then missed the other two: the first never mentioned the regular expression, the second called (?:\d+)+ "redundant" and "functionally equivalent" to \d+, which is true for what it matches and wrong for how long it takes. Neither connected int() to wrong results on ordinary sizes.

A reviewer that misses a denial of service on untrusted input is cheap for the wrong reason. If you run reviews every day, as our daily pull request review template does, pick the model before the CLI: see below.

Totals: tokens, time and cost per passed task

Over 8 runs each Claude Code (Opus 5.5) Copilot CLI (Haiku 4.5)
Passed 8/8 5/8
Total wall time 366 s 870 s
Input tokens 1.18M (92% read from cache) 6.71M (97% from cache)
Uncached input tokens 96 1,174
Tokens written to cache 88.6k (1-hour cache) 225k (5-minute cache)
Output tokens 28.4k 68.3k
Model calls / tool calls 48 / 40 215 / 216
Cost $1.50 $1.27 (127 AI credits)
Cost per run $0.187 $0.159
Cost per passed task $0.187 $0.255
Premium requests (legacy annual plan) – 2.64 (0.33 per run)

Haiku 4.5 input and output tokens cost four times less than Opus 5.5's, yet a Copilot run cost only 15% less than a Claude Code run. Copilot read 5.7 times more input and wrote 2.4 times more output, because it worked in many small steps: 215 model calls against 48. Where the money went:

Same model, different CLI: Copilot CLI running Claude

Copilot's default model on our account is a Claude model, so the cleanest way to separate the model from the agent loop was to run Claude Code on it too, claude -p --model claude-haiku-4-5, same tasks, two runs each:

Over 8 runs each, Claude Haiku 4.5 in both Claude Code Copilot CLI
Passed 5/8 5/8
Failed both reviews (1/3 each), one refactor (duplicated helper) both reviews (1/3 each), one refactor (duplicated helper)
Total wall time 696 s 870 s
Model calls / tool calls 131 / 123 215 / 216
Input / output tokens 5.16M / 81.3k 6.71M / 68.3k
Cache writes 218k at $2 per million (1 hour) 225k at $1.25 per million (5 minutes)
Cost $1.34 $1.27

The results are almost symmetrical. With the same model, Claude Code failed the same tasks in the same way: its two reviews found only the binary bug, and one of them wrote that the nested quantifier "doesn't cause catastrophic backtracking (due to anchors)", which is false; one refactor also duplicated the helper. So on these tasks, quality follows the model, not the CLI. The CLI changes the path: Claude Code made 43% fewer tool calls and finished 20% sooner, and the two ended up at about the same cost, because Claude Code's 1-hour cache writes cost 60% more per token than Copilot's 5-minute ones.

Two more data points from the same runs:

If you use Copilot, the model picker is the main quality setting. The current plans page lists Opus 5.5 for monthly plans; we could not run it, since annual plans on legacy billing don't get new models.

Pricing in practice after the switch to AI credits

Copilot plans: Free, Pro $10, Pro+ $39, Max $100, Business and Enterprise

Since June 1, 2026, Copilot usage consumes GitHub AI Credits, priced from tokens (input, output, cached) at each model's API rate. Plans on 2026-11-03:

Plan Price AI credits included each month
Free $0 Limited chat and agent use; 2,000 completions; Copilot CLI included
Pro $10 $10 base + $5 flex = 1,500 credits
Pro+ $39 $39 base + $31 flex = 7,000 credits
Max $100 $100 base + $100 flex = 20,000 credits
Business $19 per user $19
Enterprise $39 per user $39

What else to know:

Claude plans: Pro $20, Max $100, Ultra $200, Team seats, API

Plan Price Claude Code
Free $0 Not included
Pro $20/month, $17/month billed annually Included
Max $100/month Included
Ultra $200/month Included
Team Standard seat $20/month annual ($25 monthly), premium seat $100 ($125) Included
Enterprise $20 per seat per month billed annually (20 seats minimum), plus usage at API rates Included
API Opus 5.5: $4 / $20 per million input / output tokens Pay per token

Source: claude.com/pricing on 2026-11-03. Subscriptions have usage limits over a five-hour window and a week; Anthropic publishes no token count for them.

What one month of real use costs on each

Projected from our measured averages, at the prices of 2026-11-03:

Two warnings with those numbers. Copilot's cheaper run came with 3 failures out of 8: per passed task, it is the more expensive of the two. And on a Claude subscription, the same Claude Code runs cost you limits instead of dollars: none of our runs hit a limit, but nobody publishes how many tasks a five-hour window holds.

Running them unattended: headless mode and permissions

claude -p vs copilot -p: flags and output formats

Claude Code Copilot CLI
One prompt, no TTY claude -p "..." copilot -p "..."
Output --output-format json (one object) or stream-json (events) --output-format json (JSONL events), -s for the answer only
Usage in the output usage, modelUsage, total_cost_usd in the final result premiumRequests and session duration in the final result; tokens per model with --usage-output-file, and per call in ~/.copilot/session-state/<id>/events.jsonl
Model --model --model (or COPILOT_MODEL)
Spending cap – --max-ai-credits
Resume a session --resume <id>, --session-id <uuid> --resume <id>, --session-id <uuid>
Authentication in CI ANTHROPIC_API_KEY or a subscription login COPILOT_GITHUB_TOKEN, GH_TOKEN or GITHUB_TOKEN
Pin the version Install a fixed version COPILOT_AUTO_UPDATE=false

Permissions: --dangerously-skip-permissions vs --allow-all, and the silent refusal without them

We ran each CLI headless without its skip flag, in a container, with a prompt that needs a shell command (uname -s > os.txt):

Exit code File created Where the refusal shows
claude -p (no skip flag) 0 no permission_denials in the JSON (two attempts, Bash then Write)
copilot -p (no --allow-all) 0 no tool.execution_complete events with "Permission denied because no interactive user response was available"; 0.33 premium request still charged

Both exit with code 0, and both answers say politely that they could not do it. A CI job that only checks the exit code reports success for a run that did nothing. Check the denial fields, or grant exactly what the job needs: Claude Code's permission modes and allow rules, Copilot's --allow-tool, --deny-tool (which wins over --allow-all-tools) and --assisted-approval, where a model judges each request. Keep --dangerously-skip-permissions and --allow-all (alias --yolo) for disposable containers or VMs, as we did. Our post on Claude Code's --dangerously-skip-permissions tests what that flag still blocks, and lists the equivalent flag in each CLI.

Gotchas we hit

Which one should you use?

When Claude Code is the better pick

When GitHub Copilot is the better pick

Claude Code vs Copilot vs Codex vs Cursor: the short view

Claude Code Codex CLI Copilot CLI
Measured on 2026-10-06, 2026-10-13, 2026-11-03 2026-10-06 2026-11-03
Default model claude-opus-5-5 gpt-6.1-sol claude-haiku-4.5 (our account)
Passed 8/8 each time 8/8 5/8
Cost per passed task $0.181, $0.169, $0.187 $0.074 $0.255
Total wall time, 8 runs 452 s, 318 s, 366 s 389 s 870 s

Same tasks and checks on all three dates, with different CLI versions and parallelism, so compare orders of magnitude, not decimals. Codex remains the cheapest correct option on these tasks, Claude Code the fastest with the deepest reviews, and Copilot CLI depends on the model you give it. The details are in Codex vs Claude Code and Gemini CLI vs Claude Code. Cursor is an editor with its own agent rather than a CLI you script, and is not part of this benchmark.

Using Claude Code and Copilot together

The results suggest a split by task type rather than a winner: Copilot on a cheap model did well when the spec left no room for interpretation, and only Claude Code on Opus 5.5 caught every planted bug. So let Copilot implement well-specified changes, and let Claude Code review them before anything is pushed.

We ran this benchmark with plain scripts, but in SideHub the same split is a workflow: a list of steps run by an agent on your own machine, where each step can set its own CLI with provider (claude, codex, agy, copilot or gemini):

format: sidehub.workflow/v1
name: Copilot implements, Claude Code reviews
defaultProvider: claude
parameters:
  - { name: issue, label: Issue number, type: string, required: true }
steps:
  - name: Implement the issue
    provider: copilot
    prompt: |
      Read GitHub issue #{{inputs.issue}} with `gh issue view`, implement it on a new branch
      and run the test suite. Do not push.
  - name: Review the diff
    inputsFrom: ["1"]
    prompt: |
      Review the uncommitted diff of this repository for correctness, edge cases, security and
      performance. List each problem with file, line, severity and a reproducing input.
      Do not modify any file.

Each step runs as its own run, with its CLI, duration, exit code and token usage; for Copilot, SideHub reads the same ~/.copilot/session-state/<id>/events.jsonl file we used here, and shows the cost as an API equivalent. That is how to rerun this comparison on your own repository: same prompt, one step per CLI, and compare the runs. Three honest limits, all from our own use: under a per-tool approval policy, SideHub checks Claude Code's tool calls one by one but cannot do that for Copilot, so it launches Copilot without --allow-all and tools that need approval are refused; plan-limit detection relies on a Claude Code hook, while Copilot gives no structured signal; and the dead MCP server above is a Copilot problem SideHub cannot fix for you.

The template gallery has workflows to start from: Claude codes, Codex reviews is the same cross-CLI pattern with Codex, and nightly issue triage and dependency updates are unattended jobs where the cost per run adds up.

Limitations of this benchmark

The verdict

On four everyday tasks, Claude Code with Opus 5.5 passed everything and wrote the only complete reviews; Copilot CLI with its default Claude Haiku 4.5 passed 5 out of 8, for about the same price per run and 2.4 times the time. Running Claude Code on Haiku 4.5 gave the same 5 out of 8, so the comparison that matters is less "Claude Code vs GitHub Copilot" than "which model is behind it". Pick Copilot for inline completion, GitHub-native work, an annual plan that makes agent runs almost free, or well-specified changes on a cheap model; pick Claude Code for reviews, tricky changes and Opus on a $20 plan. Then rerun these four tasks on your own repository: the commands, prompts and data are all above.