Sign in

Kato Coaching

@kato-coaching.com
1.4K followers 1K following 447 posts

Turning testers into trusted advisors. | Keynote speaker | Coach | Trainer Free resources and my book: t.mtrbio.com/kato-coaching

PostsRepliesMedia
Kato Coaching @kato-coaching.com · 17/09/2026
The engineer who reads every line of every AI-written PR is now the slowest part of your release. Review metrics don't always show it. I counted:
kato-coaching.com
What AI review bots do to your review metrics
On one repository the dashboard reports six minutes to first review. The first review a person submitted took just over a day. Seven repositories, counted both ways.
000
Kato Coaching @kato-coaching.com · 16/09/2026
Tests green, a bot has reviewed it, three more waiting, so you approve it. Nothing records whether a person read it. I counted seven repos:
kato-coaching.com
What AI review bots do to your review metrics
On one repository the dashboard reports six minutes to first review. The first review a person submitted took just over a day. Seven repositories, counted both ways.
000
Kato Coaching @kato-coaching.com · 11/09/2026
1/3 "Human in the loop" gets said constantly and I could not find anyone who had measured the loop, so I counted review on seven public repositories: once as an indiscriminate dashboard query would, once counting only what a person submitted.
100
Kato Coaching @kato-coaching.com · 10/09/2026
1/4 I counted code review twice on seven public repositories: once the way a dashboard counts it, once counting only the reviews a person submitted. Eight weeks of merged pull requests. The two counts are a long way apart.
100
Kato Coaching @kato-coaching.com · 17/07/2026
Asked AI for 30 rows of signup test data. Got 30 tidy, valid, useless rows. Treating it as a pair instead of an oracle, constrain, generate, verify, surfaces the bug the happy-path data missed. Two-minute demo on a real form.  #softwaretesting 
youtube.com
Stop Treating AI Like an Oracle: Better Test Data in Two Minutes
If AI keeps handing you clean, useless test data, that's down to ho...
000
Kato Coaching @kato-coaching.com · 10/07/2026
The argument for not testing a prototype is that you don't know if it's worth keepin. That's no longer true.  I'm partnering with SmartBear on a sponsored series of content about tools for the AI era. Opinions are entirely my own.
kato-coaching.com
When Testing Costs an Hour, Not Days - Kato Coaching
I’m partnering with SmartBear on a sponsored series of content about tools for the AI era. As always, all opinions are entirely my own. A while ago I watched a mobile app go from a sales prop to a production system in the space of a single meeting. It had been built as a demo, […]
000
Kato Coaching @kato-coaching.com · 19/06/2026
Buying a testing agent: easy. Knowing if it works, recalibrating it, watching its reasoning, drawing its authority line: not in the box. The five questions I'd start with: kato-coaching.com/fighting-ai-with-… Paid partnership w/ SmartBear.
kato-coaching.com
Fighting AI with AI is the easy part - Kato Coaching
I’m partnering with SmartBear on a sponsored series of content about tools for the AI era. As always, all opinions are entirely my own. SmartBear ran a survey in January 2026 of 273 software testing and quality decision-makers, mostly directors and above, and the writeup, their AI Software Quality Gap Report, lets the numbers speak […]
010
Kato Coaching @kato-coaching.com · 18/06/2026
Holy guacamole. Long-ish read, pretty jaw dropping. bobdahacker.com/blog/fifa-hack
bobdahacker.com
I Could've Rickrolled the Entire FIFA World Cup. All I Needed Was My ID.
How I found that anyone could register on FIFA's public Agent Platform, gain access to the Football Data Platform's Streaming Management panel, and get RTMP ingest URLs and stream keys for every live ...
010
Kato Coaching @kato-coaching.com · 17/06/2026
You can't manually test AI-generated code at AI-generation speed. That math isn't mathing. 40% of teams generate 41%+ of their code with AI. 92% still test by hand. Source:
smartbear.com
AI Software Quality Gap Report | SmartBear
Explore how AI is reshaping software quality. Get insights on testing gaps, risk, and how teams can maintain control as development accelerates.
000
Kato Coaching @kato-coaching.com · 11/06/2026
Everyone in QA can describe their AI's bad habits without thinking. Nobody finishes "what I really want from this tool is…" If you've been generating tests with AI and correcting the same handful of things every week, what would actually help?
011
Kato Coaching @kato-coaching.com · 10/06/2026
The research on code review has been consistent for over a decade: it's strong at knowledge sharing, weak at catching defects. When teams insist on "humans review every line" of AI output, they're doubling down on the thing we know doesn't work. kato-coaching.com/human-in-the-loop-code-review
010
Kato Coaching @kato-coaching.com · 02/06/2026
The AI keeps making the same test mistake because the decision to stop is buried in a Slack thread it's never read. Write the rule. Put it in the file your AI reads at generation time. Walkthrough: youtu.be/ccFaZdGUSkw #AIinTesting #QA
youtu.be
Where AI test rules live (and how to write one that works)
Your AI keeps making the same test mistake. You keep leaving the sa...
130
Kato Coaching @kato-coaching.com · 07/05/2026
> 65% of QA teams describe AI output as "noisy, incorrect, or requiring rework." > > Mapping where the verification time actually goes. 5-min survey, no opt-in: >
docs.google.com
AI Playbook Course: Pre-Launch Survey
Five questions, three minutes. Your answers shape the next AI Playbook cohort.
020
Kato Coaching @kato-coaching.com · 06/05/2026
Last night Alan and I talked about the new cool kid on the block, and how it's changing testing. Or maybe not?  I'll link Apple podcasts, but it's available everywhere you get your podcasts. 
podcasts.apple.com
Episode 230: Kat Obring is Back
Podcast Episode · AB Testing · May 5 · 54m
000
Reposted by Kato Coaching
Jess Rose @jesslynnrose.bsky.social · 24/04/2026
Our free web development bootcamp launch party is today! Join: @vivaldi.com's @brucel.bsky.social will be showing us the lovely side of HTML Me for how good notes drive learning @freecodecamp.bsky.social's @quincy.bsky.social for a fireside chat about learning We would love to see you there!
badwebsite.club
Launch Party
Virtual kickoff party to start the bootcamp. You don’t need to prepare anything, but you may want to come prepared to take notes. We’ll have several guests, including: freeCodeCamp Founder Quincy Laws...
1117
Kato Coaching @kato-coaching.com · 24/04/2026
It’s no secret that I love working with claude code. If you’ve not tried it yet but would like to: turns out I’ve got three passes for a free week of it. Grab it here claude.ai/referral/WY0...
claude.ai
Join Claude!
You've been invited to try Claude
000
Kato Coaching @kato-coaching.com · 23/04/2026
I have a Claude Code skill that creates google slides for me, and then adds speaker notes. Since the update to Sonnet 4.7 it constantly "forgets" how to do the latter, and tells me it's incapable of it. Anybody else have this problem? Suggestions?
100
Kato Coaching @kato-coaching.com · 13/04/2026
The "testing mindset" isn't a personality trait. It's trained. Developers often don't think about quality the way QA engineers do because they haven't had the same deliberate practice. That's fixable, but only if the culture reinforces it too.
020
Kato Coaching @kato-coaching.com · 09/04/2026
Before you conclude AI isn't worth it, measure how long correction actually takes. One experiment, a real task, less than a day.  Free guide: resources.kato-coaching.com/5-ai-experiments #SoftwareTesting #QA #AIinTesting
000
Kato Coaching @kato-coaching.com · 02/04/2026
The Easter Bunny has 10,000 eggs to hide across 47 gardens before sunrise. His team suggests using AI to plan the hiding spots. Before handing a task to AI, run it through a simple framework. Here is how the Easter Bunny uses this framework to avoid a disaster. 🐇🥚
youtu.be
Can AI save the Easter Bunny?
Before you hand a task to AI, you need to know if it's actually a good candidate. The Easter Bunny has 10,000 eggs to hide across 47 gardens before sunrise. ...
000
Kato Coaching @kato-coaching.com · 31/03/2026
Getting the task type wrong makes the model choice irrelevant. The scorecard tells you what kind of solution actually fits before you commit to one.  resources.kato-coaching.com/scoreca… #SoftwareTesting #AItools #QA
resources.kato-coaching.com
The AI Use Case Scorecard (Free)
Most AI output problems start before you open the tool. This free scorecard helps you check before you spend hours correcting output.
110
Kato Coaching @kato-coaching.com · 31/03/2026
TIL: it's much easier to get Claude Code to play nicely with the Google workspace MCP than it is to get Gemini to use it. What the hell? Yes, I am experimenting with a multiple agent setup, but man, it's painful.
020
Kato Coaching @kato-coaching.com · 30/03/2026
AI gave you 12 scenarios. Now what? The list is raw material, not a test plan, and your judgment is what turns it into one. New video on how to triage what AI gives you: youtu.be/0awmamL1l2c #softwaretesting #AIinTesting
youtu.be
How to Evaluate AI-Generated Test Scenarios
AI can give you a decent list of test scenarios - if you prompt it well. But the list is starting material, not a test plan. The work of deciding which scena...
000
Kato Coaching @kato-coaching.com · 27/03/2026
Agile gave us JIRA. DevOps gave us pipelines. Neither answered the harder question. AI evals is doing the same thing: the tooling is excellent. The question underneath it remains.  kato-coaching.com/the-hard-part-of-… #AITesting #QualityEngineering #SoftwareTesting
kato-coaching.com
The Hard Part of AI Evals Isn't the Tooling - Kato Coaching
Additional thoughts on session three of the AI Evals and Analytics Playbook1. The first part is here2. Every major shift in how we build and ship software has been followed by a wave of tooling that automates the tractable part and leaves the actual problem to the practitioner. Agile gave us story pointing ceremonies and […]
000
Kato Coaching @kato-coaching.com · 26/03/2026
Most AI pilots produce impressions, not evidence. An acceptable error rate isn't a feeling. It's a number you write down before the experiment starts.  resources.kato-coaching.com/scoreca… #SoftwareTesting #AItools #QA
resources.kato-coaching.com
The AI Use Case Scorecard (Free)
Most AI output problems start before you open the tool. This free scorecard helps you check before you spend hours correcting output.
110
Kato Coaching @kato-coaching.com · 25/03/2026
"The profession keeps arriving at “quality is systemic” as though it’s a fresh revelation, every time a new technology makes the verification frame obviously inadequate. It happened with continuous delivery. [...] It is happening now with AI agents."
kato-coaching.com
The Verification Trap - Kato Coaching
Someone posted a thinking piece in a Slack channel I’m in last week, long and earnest and well-structured, arguing that quality engineering needs to evolve for the age of agentic AI, that we need to stop thinking about quality as testing and start treating it as a systemic property. I read it and felt a […]
010
Kato Coaching @kato-coaching.com · 24/03/2026
Every AI demo works because the vendor picked the use case. When you're back at your own codebase, the conditions are different. Define your problem before you evaluate the tool.  resources.kato-coaching.com/scoreca… #SoftwareTesting #AItools #QA
resources.kato-coaching.com
The AI Use Case Scorecard (Free)
Most AI output problems start before you open the tool. This free scorecard helps you check before you spend hours correcting output.
010
Kato Coaching @kato-coaching.com · 23/03/2026
The vague prompt gets you a generic list. Two minutes of context - scope, users, risk angle - gets you scenarios worth thinking about. Short video on how to close that gap.  youtu.be/55SLTvKq7rE #softwaretesting #AIinTesting
youtu.be
How to write AI prompts that give you useful test scenarios
Most AI prompts for testing are vague — and you get vague output back. "Write test cases for the SplitPay feature" returns a generic list you already knew. N...
000
Kato Coaching @kato-coaching.com · 20/03/2026
The AI evals field calls it a rubric. If you've done serious testing work, you probably know it by a different name.  kato-coaching.com/the-anatomy-of-a-… #SoftwareTesting #AITesting
kato-coaching.com
The anatomy of a metric - Kato Coaching
Session two closed with a question I couldn’t answer.1 When a product scores 78% on a given metric, what tells you whether that’s good enough to ship? I flagged it as something session three would probably address, and it did, though not in the way I expected. The question can’t be meaningfully answered until you’ve […]
000
Kato Coaching @kato-coaching.com · 19/03/2026
When AI adoption is handed to IT, you get a deployment. Deciding what good output looks like and how to verify it belongs to the people doing the work. resources.kato-coaching.com/scoreca…  #QualityEngineering #TechLeadership
resources.kato-coaching.com
The AI Use Case Scorecard (Free)
Most AI output problems start before you open the tool. This free scorecard helps you check before you spend hours correcting output.
000
Kato Coaching @kato-coaching.com · 18/03/2026
When teams say the AI tool isn't working, I ask about the process before the AI arrived. Usually the requirements were always vague. The tool is just making that visible. resources.kato-coaching.com/scoreca… #SoftwareTesting #AIinTesting #QA
resources.kato-coaching.com
The AI Use Case Scorecard (Free)
Most AI output problems start before you open the tool. This free scorecard helps you check before you spend hours correcting output.
000
Kato Coaching @kato-coaching.com · 17/03/2026
Six months into an AI pilot with no success criteria is just six months of accumulated impressions.  You can't pass or fail a hope. 
resources.kato-coaching.com
The AI Use Case Scorecard (Free)
Most AI output problems start before you open the tool. This free scorecard helps you check before you spend hours correcting output.
000
Kato Coaching @kato-coaching.com · 13/03/2026
I am working on a food tracker for my very specific needs, and Claude just turned into every developer I ever worked with.
011
Kato Coaching @kato-coaching.com · 10/03/2026
"The hard part is scalability, not automation." That line from session one of "AI evals and analytics" confused me. Session two explained it.  Full write-up in my blog: kato-coaching.com/the-ai-evals-fiel… #AIEvals #SoftwareTesting #QA
kato-coaching.com
The AI evals field chose a flawed tool and stuck with it - Kato Coaching
Session one left me with two things I hadn’t resolved.1 The first was a line the instructor said almost in passing: “the hard part is scalability, not automation.” I wrote it down because it piqued something, but I couldn’t quite work out what problem it was pointing at. The second was a question I kept […]
000
Kato Coaching @kato-coaching.com · 06/03/2026
Updated my website this week — it should finally be clear what I do and how to work with me. Courses, workshops, and 1:1 coaching for QA professionals.  kato-coaching.com
000
Kato Coaching @kato-coaching.com · 05/03/2026
If the output is slop regardless of how you phrase it, the problem isn't the prompt. It's the use case.  Free scorecard:
resources.kato-coaching.com
The AI Use Case Scorecard (Free)
Most AI output problems start before you open the tool. This free scorecard helps you check before you spend hours correcting output.
010
Kato Coaching @kato-coaching.com · 04/03/2026
Before I wrote today's post, I defined what good looked like: gives value, doesn't rely on outrage, sounds like me. Writing to a clear brief changes the experience. So does diagnosing a draft. Testers do this before they run anything.
000
Kato Coaching @kato-coaching.com · 03/03/2026
The AI correction loop usually starts before the tool is opened, at the moment someone chose the wrong use case for it. Testers already know how to ask whether a tool suits a problem. That skill just hasn't been applied here yet. More soon.
000
Kato Coaching @kato-coaching.com · 27/02/2026
The skills QA professionals already have (defining success criteria, testing behaviour, not trusting metrics at face value) are exactly what's missing from most AI integrations. I'm learning AI evals to understand why.  Session one: kato-coaching.com/what-i-dont-understand-about-ai-evals-yet/
100
Kato Coaching @kato-coaching.com · 26/02/2026
Ran my workshop "Deciding Fast" on Tuesday with a software team in Sweden. Everyone in one room sharing computers, no breakouts. It's built for remote. I adapted. 6/8 Good or Excellent. Best response to "what will you do differently?": "Set clearer success condition." #SoftwareTesting #AITesting
000
Kato Coaching @kato-coaching.com · 25/02/2026
"Tell me what you're uncertain about." "Push back if my assumptions are wrong." Only 30% of people give instructions like these. The model defaults to confident and agreeable.  www.anthropic.com/research/AI-fluen… #SoftwareTesting #AITesting #AILiteracy
anthropic.com
Anthropic Education Report: The AI Fluency Index
Anthropic's AI Fluency Index measures 11 observable behaviors across thousands of Claude.ai conversations to understand how people develop AI collaboration skills.
000
Kato Coaching @kato-coaching.com · 24/02/2026
The strongest predictor of AI fluency, per Anthropic's research: iteration. Treating the first response as a draft, not an answer. 5.6x more likely to question reasoning. Familiar territory if you work in testing.  www.anthropic.com/research/AI-fluen… #SoftwareTesting #AITesting
anthropic.com
Anthropic Education Report: The AI Fluency Index
Anthropic's AI Fluency Index measures 11 observable behaviors across thousands of Claude.ai conversations to understand how people develop AI collaboration skills.
000
Kato Coaching @kato-coaching.com · 19/02/2026
Six months in and nobody can say whether the AI is actually working. Not anecdotally, but with evidence. That gap is the most common thing I see in QA teams right now.  How do you measure if the new licence is worth the money? #softwaretesting #QA #AItools
020
Kato Coaching @kato-coaching.com · 18/02/2026
18% of testers I surveyed said their top AI frustration is not bad output. It is that the tools have no sense of test strategy. The AI is not wrong. It is indiscriminate. #AITesting #SoftwareTesting #TestStrategy
000
Kato Coaching @kato-coaching.com · 17/02/2026
I timed myself running tests manually, then gave the same task to an AI tool and timed that too. The result was not what I expected. Full breakdown with video later this week.
110
Kato Coaching @kato-coaching.com · 13/02/2026
You ask the AI to generate a few test cases for a feature you know inside out. Save yourself twenty minutes. What comes back looks reasonable. Then you start reading. Read the full story in my blog:
kato-coaching.com
You were promised speed. You got a new job instead. - Kato Coaching
You know the feature inside out. You have tested it, broken it, rebuilt the test suite around it twice. So you ask the AI to generate a few test cases. Save yourself twenty minutes. What comes back looks reasonable. The structure is right. The naming conventions are close enough. Then you start reading. The first […]
000
Kato Coaching @kato-coaching.com · 12/02/2026
Why do so many testers call AI "exhausting"? It is not the learning curve. It is the correction loop. You spend 45 minutes fixing output that was supposed to save you an hour. Five experiments to figure out which tasks AI actually improves:
resources.kato-coaching.com
5 AI Testing Experiments You Can Run This Week (Free Guide)
65% of testers say AI output is unreliable. This free guide gives you 5 small experiments to figure out where AI actually helps your testing workflow.
000
Kato Coaching @kato-coaching.com · 11/02/2026
I asked testers what worried them most about AI. 24% said management expectations, not the tools. "Pressure to deliver has increased dramatically because management thinks we must be twice as productive now." Argue with evidence:
resources.kato-coaching.com
5 AI Testing Experiments You Can Run This Week (Free Guide)
65% of testers say AI output is unreliable. This free guide gives you 5 small experiments to figure out where AI actually helps your testing workflow.
010
Kato Coaching @kato-coaching.com · 10/02/2026
I surveyed 17 testers about their biggest AI frustrations. 65% said the same thing: the output is unreliable. One called it "slop." Another described the correction loop as exhausting. I wrote a free guide: 5 small experiments
resources.kato-coaching.com
5 AI Testing Experiments You Can Run This Week (Free Guide)
65% of testers say AI output is unreliable. This free guide gives you 5 small experiments to figure out where AI actually helps your testing workflow.
000
Kato Coaching @kato-coaching.com · 06/02/2026
I have blogged about why I think the rise of AI tooling is not the end of the world, as many in the testing community see it. 
kato-coaching.com
Why the bottleneck was never just about thinking - Kato Coaching
I’ve been seeing a lot of posts on LinkedIn recently that make the same claim: speed of writing code was never the bottleneck in software engineering. The real bottleneck was always thinking, choosing the right thing to build, understanding the problem. AI has just made that painfully obvious. I partly agree, but the framing is […]
000