Toolkit
← All posts

Using AI to write code: what actually works

Toolkit Blog · · AI & coding · 9 min read

As of mid-2026, roughly 90% of developers report using AI at work, and more than 80% believe it makes them more productive[3] — yet the best controlled study we have found experienced developers were actually slower with AI on their own codebases, while believing the opposite.[2] Both things can be true. AI coding tools are genuinely powerful and genuinely easy to use badly. This piece is about the difference: where the tools earn their keep, where they burn your time, and the working habits that separate developers who ship faster from developers who just feel faster.

Where AI genuinely shines

Some tasks are almost embarrassingly well-suited to a language model. The common thread: the task is well-specified, the pattern exists in a million training examples, and you can check the result quickly.

Notice what's on the list: tasks where verification is cheap. That's not a coincidence, and it's the lens for everything that follows.

Where it quietly wastes your time

In 2025, the research group METR ran a randomized controlled trial that should hang over every AI-coding discussion: 16 experienced open-source maintainers completed 246 real issues in mature repositories they knew deeply, with tasks randomly assigned to allow or forbid AI tools. With AI, they took 19% longer. They had predicted AI would make them 24% faster — and even after finishing, still believed it had made them about 20% faster.[2]

METR is careful about the caveats, and so should you be: this was a snapshot of early-2025 tools, in one demanding setting — experts on large codebases they knew inside out — and it says nothing about other developers or later tools.[2] But the perception gap is the durable lesson. AI assistance feels fast because the keystrokes stop being yours. Whether it is fast depends on how long you spend steering, reviewing, and repairing plausible-but-wrong output.

The time sinks cluster predictably. Code that depends on deep, undocumented context the model doesn't have. Iterating on vague prompts instead of thinking for two minutes first. Debugging a subtly wrong 300-line generation instead of writing a correct 60-line version yourself. If you know exactly what a small diff should be, typing it is often the fast path — Anthropic's own guidance says to skip the planning ceremony when you could describe the change in one sentence.[1]

You own every line

The single non-negotiable rule: never merge code you haven't read and understood. Not skimmed — read. "The AI wrote it" will never appear in a post-incident review as an excuse; the commit has your name on it, and so does the pager.

The industry-scale data backs the discipline. Google's 2025 DORA report found AI adoption now correlates with higher software delivery throughput — but still with worse delivery stability, because more change volume exposes every weakness downstream of it. Its summary line is the most useful sentence written about this technology: AI doesn't fix a team, it amplifies what's already there.[3] Teams with strong review culture, automated tests and fast feedback loops get faster. Teams that rubber-stamp PRs get broken releases, faster. Notably, about 30% of developers surveyed report little or no trust in AI-generated code[3] — a healthy instinct, as the security numbers below confirm.

Practical review habits that hold up:

  1. Review the diff, not the vibe. Read generated changes with the same hostility you'd give a stranger's pull request. Small, reviewable increments beat one giant generation.
  2. Make the model explain itself. Ask why it chose an approach, what breaks it, which edge cases it handles. Wrong answers here are your earliest warning.
  3. Watch for plausible-looking APIs. Hallucinated methods and invented package names still happen; they compile in the model's imagination only. Every import earns an existence check.
  4. Keep commits separable. If an AI-assisted change goes bad in production, you want to revert one commit, not archaeologically excavate it from a week of mixed work.

Review tip: when a model rewrites a file, don't trust your eyes to spot what changed — paste the before and after into the diff checker and read the actual delta. It runs entirely in your browser, so your code never leaves your machine.

Context is most of the job

The quality gap between developers using the same model is mostly a context gap. A model can't read your mind, your architecture docs, or your team's taste — unless you hand them over. Anthropic's best-practices guide for Claude Code is blunt about it: the more precise your instructions, the fewer corrections you'll need.[1] The same guidance applies to any assistant or agent:

Make it prove its work

A language model's output always looks done — that's precisely what it's optimized for. So looking done must never be the acceptance test. Give the model a check it can run: a test suite, a build, a linter, a script that diffs output against a known-good fixture. With a pass/fail signal in the loop, the model iterates against reality instead of against your patience; without one, you personally are the verification loop, and every mistake waits for you to notice it.[1]

The strongest version is test-first: write (or dictate) the tests yourself, then let the model make them pass. Tests you authored encode your understanding of the problem, which is exactly the thing you're otherwise trusting the model to guess. And demand evidence over assertion — actual test output, the command it ran and what it returned — because reviewing evidence is faster than re-running everything yourself.[1] One warning: a model told to make tests pass will sometimes take shortcuts, special-casing the test's inputs or weakening an assertion. Generated changes to test files deserve your most suspicious read.

The habit generalizes into a rule short enough to remember: if you can't verify it, don't ship it.[1]

The security and license fine print

Here the data is genuinely uncomfortable. Veracode's 2025 GenAI Code Security Report tested over 100 LLMs on 80 curated coding tasks across Java, Python, C# and JavaScript: 45% of the time, models introduced an OWASP Top 10 vulnerability when a secure and an insecure implementation were both available. For cross-site scripting defenses the failure rate hit 86% of relevant samples; Java fared worst of the languages at a 72% failure rate. Worse, security performance stayed flat as models got newer and larger — the functional code got better, the secure code didn't.[4]

The mechanism is mundane: models reproduce the average of their training data, and the average public code sample handles input sanitization, secrets and authentication carelessly. Unless your prompt makes security requirements explicit, the model decides for you — so state them (parameterized queries, output encoding, no secrets in code), run static analysis on generated code just as you would on human code, and give extra scrutiny to anything touching auth, file paths, SQL or user input.

Licensing is the quieter pitfall. Generated code can occasionally match public code verbatim — GitHub says matches occur in under 1% of Copilot suggestions, and its code-referencing feature checks suggestions against an index of public repositories and surfaces the matching file and its license, with a policy option to block such suggestions outright.[5] Under 1% is small until you multiply it by thousands of suggestions a week. If you ship commercial software, turn these controls on; a copyleft-licensed snippet pasted into proprietary code is a problem no model will flag on its own.

Skills atrophy, and when to leave AI out

The honest version of the atrophy concern: it's real, but it's not automatic. A CHI 2025 study of 319 knowledge workers by Microsoft Research and Carnegie Mellon found that higher confidence in the AI was associated with less critical thinking, while higher confidence in one's own skills was associated with more — and that AI shifts the thinking that remains toward verifying, integrating and stewarding output rather than producing it.[6] The muscle you stop using is the one you lose; the developers who stay sharp are the ones who keep treating the model's output as a claim to evaluate, not an answer to accept. Deliberate practice still matters — sometimes write it yourself, on purpose, especially in the areas you're supposed to be the expert.

And some work you should simply keep away from the tools, as of mid-2026:

The through-line of all of it: AI coding tools reward exactly the practices that made software good before them — clear specs, small reviewable changes, strong tests, healthy suspicion. That's the amplifier effect in DORA's data[3], and it's the cheerful conclusion hiding inside the scary studies. The tools don't replace engineering judgment. They pay interest on it.

Sources

  1. Anthropic — Best practices for Claude Code (official documentation), accessed July 2026
  2. METR — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, July 2025, accessed July 2026
  3. Google Cloud — Announcing the 2025 DORA (State of AI-assisted Software Development) Report, accessed July 2026
  4. Veracode — Insights from the 2025 GenAI Code Security Report, accessed July 2026
  5. GitHub Docs — GitHub Copilot code referencing, accessed July 2026
  6. Microsoft Research — The Impact of Generative AI on Critical Thinking (CHI 2025), accessed July 2026

Related: Why Toolkit runs entirely in your browser (and why that matters) · Diff checker · Browse all tools