Token cost: is it cheaper to let an agent manage Kubernetes with CLI or MCP?

The short answer#

In forty measured agent runs on Kubernetes operations, the plain CLI setup used 76% fewer tokens per successful task than the MCP setup, and it succeeded more often: 10 out of 10 against 7 out of 10. All three CLI-based setups landed 76–77% below MCP.

The saving did not come from a clever prompt or an output-compression tool. It came from measuring the right unit, tokens per verified success, and moving away from the toolchain that looked fine on the surface but carried almost three times the context into every model call.

The experiment: every task, every toolchain, twice

Raw CLITargeted CLIKubernetes MCPOutput compression

Multifault incident

Failed rollout

Capacity change

Memory incident

Helm lifecycle

5 tasks × 4 toolchains × 2 runs = 40 measured runs · 729 model calls

What happened in every run

1. 01ResetA trusted runner resets the fault fixture and records the baseline.
2. 02RunOne isolated agent session with its own workspace and scoped credentials.
3. 03CheckAn independent evaluator verifies specs, readiness and a real checkout.
4. 04MeasureAgent Tokenwatch captures the session; counts are reconciled per response.

Run it yourself: the exact steps#

The unit that matters is tokens per task that actually passed, with failed attempts left in the cost. A monthly total hides both an agent that is cheap per call but needs more calls and one that is cheap because it quietly did the job wrong. These are the steps and Agent Tokenwatch commands the study used to get that number. Tokenwatch stores only token metadata, locally, never prompts, code or tool output.

1. Install Tokenwatch. The study pinned version 0.2.0.

npm install -g agent-tokenwatch

2. Write the experiment down before collecting anything. The journal entry holds the hypothesis, the change, the baseline, the metric, the guardrail and the rollback, so the goalposts cannot move once the numbers arrive. This is the study’s first entry, shortened:

tokenwatch experiment add \
  --hypothesis "Targeted CLI guidance reduces input+output tokens per independently successful task versus raw CLI" \
  --change "Raw CLI vs targeted CLI instructions; Kubernetes MCP secondary; same model, permissions and fixture resets" \
  --baseline "Raw CLI arm, run in the same cohort" \
  --metric "Input+output tokens of all attempted runs / independent successes, per toolchain" \
  --guardrail "Success rate must not drop; failed runs stay in the cost" \
  --rollback "Keep raw CLI as the default" \
  --sample "5 tasks x 3 toolchains x 2 repetitions"

It prints the experiment’s id, exp-2026-10-02-75cc3e in the study.

3. Give every run its own store and its own agent configuration. Each run then holds exactly one agent session, kept apart from the coordinating session and from anything else on the machine:

RUN=$PWD/runs/raw-cli/multifault-r1
mkdir -p "$RUN"/{work,codex-home,tokenwatch} && chmod 700 "$RUN"/tokenwatch
export TOKENWATCH_HOME="$RUN/tokenwatch" CODEX_HOME="$RUN/codex-home"
tokenwatch install --agents codex --scope project --project "$RUN/work" \
  --codex-config "$RUN/codex-home/config.toml"

The --codex-config flag matters: without it the install writes to your default agent configuration and the isolation is gone. If you start Codex through your own launcher (the study used one for its local Qwen model), point Tokenwatch at it with tokenwatch config set codex.command /absolute/path/to/launcher.

4. Run the agent through the Tokenwatch wrapper, one run at a time. A session started as plain codex records no tokens. Runs share the local telemetry receiver, so they must not overlap:

START=$(date -u +%Y-%m-%dT%H:%M:%SZ)
tokenwatch-codex exec --json --skip-git-repo-check \
  --cd "$RUN/work" "$(cat prompts/multifault.md)"

5. Check success yourself, never by the agent’s word. In the study a separate evaluator compared the specification, checked readiness and walked a real checkout. That verdict, not the agent’s report, decides what counts as a success.

6. Export exactly that session, then check it.

tokenwatch doctor --json > "$RUN/doctor.json"
tokenwatch analyze --since "$START" --agent codex \
  --group-by session --limit 100 --format json > "$RUN/usage.json"

The export must contain one session, the run’s own. I also reconciled every export against the agent’s own usage records, response by response, before trusting it. Count total tokens as input plus output; cached input is part of input, counted once. Never use global dashboard totals: they include sessions you are not measuring.

7. Divide and close the experiment with a verdict. Tokens per success is the sum over all attempted runs of a toolchain divided by its verified successes. Then record what the evidence supports, adopted, rejected or inconclusive:

tokenwatch experiment close exp-2026-10-02-75cc3e --result inconclusive \
  --outcome "Raw CLI 10/10, 3.61M tokens; targeted CLI 10/10, 3.57M (-1.3%), but -38% then +45% between rounds; MCP 7/10, 10.67M"
tokenwatch experiment list --all

I closed both experiments, this one and the later output-compression one, inconclusive, because the evidence did not support a savings claim. The same loop works on everyday work: pick one change you believe saves tokens, write it into the journal, run the same kind of task before and after, more than once, and keep the change only if tokens per verified success fall.

The experiment#

I ran a coding agent (Codex on a local Qwen model) against a running Online Boutique application on Kubernetes, with five DevOps tasks:

• diagnose and repair a combined routing and dependency incident
• recover a failed release
• apply a planned capacity change
• fix a memory-limit incident
• install a Helm chart with requested values, keep it healthy, then uninstall and clean up

The test bench

One Windows PC · Docker Desktop (WSL2) · NVIDIA GPU · one shared Docker network

• devcontainerAgent sandbox
  • Codex CLI, launched by a trusted runner, one run at a time
  • Agent Tokenwatch: capture wrapper and local metadata ledger
  • kubectl · Helm · Kubernetes MCP server · output compression tool
  • Independent evaluator: specification, readiness, real checkout
• containerLocal model
  • llama.cpp serving Qwen 3.8 27B on the local GPU
  • 245k-token context window
  • No cloud model calls, no API bill
• containerKubernetes (k3s)
  • Online Boutique, a microservice web shop
  • 5 tasks: multifault incident, failed rollout, capacity change, memory incident, Helm lifecycle
  • Fault fixture reset and baseline before every run
• one folder per runIsolated runs
  • runs/<toolchain>/<task>-r<n>/
  • toolchains: raw-cli · targeted-cli · kubernetes-mcp · compression
  • each run: its own work/, codex-home/, tokenwatch/ and artifacts/
  • scoped Kubernetes credentials per run

• Codex→Qwenmodel calls through a local compatibility proxy
• Codex→Tokenwatchtoken counts over local OpenTelemetry, no content
• kubectl · Helm · MCP→k3sthe agent's only way into the cluster

Each task ran twice with each of four ways of giving the agent access to the cluster:

Toolchain	What the agent got
Raw CLI	kubectl and helm
Targeted CLI	the same tools, plus guidance to scope queries and limit output
Kubernetes MCP	native MCP server tools
Output compression	kubectl and helm behind a stock compression tool, run later

Every run started from a reset fixture with its own workspace and scoped credentials, and the same diagnose → repair → report process. Success was strict and checked independently: the right specification with no unrelated changes, workloads ready, and a fresh customer journey from browsing to checkout. For Helm, the values actually stored in the release counted, not just healthy pods.

The results#

Toolchain	Strict success	Tokens, all attempts	Tokens per success
Raw CLI	10/10	3.61M	361k
Targeted CLI	10/10	3.57M	357k
Kubernetes MCP	7/10	10.67M	1,525k
Output compression	10/10	3.45M	345k

Tokens per verified success

• Raw CLI361k
• Targeted CLI357k
• Kubernetes MCP1,525k
• Output compression345k

Per verified success, the raw CLI used 76% fewer tokens than the MCP setup. That is the saving worth having.

Where the 76% came from#

The MCP setup was containers/kubernetes-mcp-server v0.0.67. Its runs did not make many more model calls: 195 against 184 for the raw CLI. Each call was simply much heavier, about 55,000 tokens on average against 20,000. Every turn re-reads the conversation so far, so whatever the tools put into the context is paid for again on every later call.

Two things made the MCP context heavier. The first is the tool descriptions. The prompts were identical across toolchains, yet the very first MCP call already carried about 9,200 tokens against 5,000 for the raw CLI. That gap of about 4,200 tokens is the MCP tool schemas, and they ride along on every call: about 0.8M tokens over the 195 calls, roughly an eighth of the extra cost. The second, and larger, is the tool results. The MCP tools returned about 11 times more data (3.56 MB across 310 tool calls, against 0.31 MB across 195 for the raw CLI). The agent cut each result off at about 50 KB, so about 4 times more actually reached the model: 1.41 MB against 0.33 MB. The study’s own reports do not break the cost down this way; these figures are my reading of the raw session streams.

Then the failures. Three MCP runs restored a healthy-looking application but broke the task contract: two Helm installs ended with the wrong stored values, and one rollout added an unrequested port-name change. The Helm values arrived at the server empty, most likely because the tool schema declared the values parameter without any properties; the server accepted the same values when called directly, so this points at the integration rather than at Helm. Their tokens stay in the cost, and they count as failures. A check that only asks “is the app up?” would have scored all forty runs a success.

This is a result for one MCP integration with one model, not a verdict on MCP in general. It is exactly why a team should measure its own setup instead of choosing one on reputation.

What did not save tokens#

Scoped-query guidance saved 1.3% overall. Summed over the five tasks in each round, it saved 38% in the first round, then used 45% more in the second. The same task with the same tools varied about as much between two runs as the two toolchains varied from each other. A single before-and-after comparison would have “proven” whichever story you wanted.

Output compression came in 4.6% below the earlier raw-CLI runs per success. When I checked the mechanism, it had removed 413 bytes out of 364,271 captured: about a tenth of a percent. None of its filters matched these commands. The small difference is more plausibly explained by the agent taking slightly different paths, the cache state and the runs happening at a different time; the study could not separate these. Installing an optimization is not evidence that it worked.

Why: consumption is context, not output#

Across all four setups, about 95% of the input was cached context the model had already seen, re-read on every turn. The model’s own output was only 2–4% of the tokens. What tools print goes into that context, so it is part of the large slice and is paid for again on every later turn. That is why output compression is a sound idea in principle; here it simply removed only a tenth of a percent. The levers that worked are the ones that keep the context small: lean tools, no giant results, and fresh sessions for fresh tasks.

A playbook for a DevOps team#

1. Instrument per session. Capture tokens for each agent session, not as a monthly total.
2. Define what success means before you measure. Write the contract down: specification, readiness, a real user journey, and the details that count, like the values Helm actually stored.
3. Count failures in the cost. Tokens per verified success, never tokens per attempt.
4. Compare toolchains on the same tasks, at least twice each. Expect differences of tens of percent between repetitions; do not trust single runs.
5. Check the mechanism before crediting an optimization. If it claims to compress, measure what it removed.
6. Watch tokens per call. A jump in per-call size points at tool descriptions and results swelling the context.

The limits, plainly#

Five tasks, one application, one local model, two repetitions each. The output-compression arm ran later than the others, so it is a historical comparison, not a side-by-side one. The model ran locally, so these are model-reported tokens, not a bill or an energy figure. Shared inference cache, host load and treatment order were not fully controlled. And the governance benefits sometimes claimed for MCP were not tested here. It is a small, honest dataset, and the method matters more than any single number in it.