By Kjell Omdal Erichsen, Testify
I’ve been using AI tools in my daily work for about twenty months now. Not to replace testing, but to do more of it, faster, in situations where the alternative was simply not testing at all. Here are three short stories to illustrate what that actually looks like.
#1 The integration problem
Working with clients on complex integration landscapes, the bottleneck is rarely the testing itself. It’s everything around it: getting the right test data, setting up the right environment state, wiring together the right tools quickly enough that the window of opportunity doesn’t close before you’ve started.
When a test lead came to me one morning in March, he was frustrated. The legacy database QA environment had been down for two weeks for an upgrade. He had no test data. His project was blocked. The question wasn’t “How do we design a great test suite?”, it was rather “How do we get anything running at all today? We really needed it a week ago”
I sat down next to him, opened Claude, and we started building. Within a few hours we had a working tool that took CSV exports from a legacy database and published them as messages on an Azure Service Bus topic, using exactly the format the system under test expected. The next day he needed a different message shape. We extended it. The day after that, a third variant. By the end of the week he had a single-click desktop application he could run himself, logged in with his own credentials, no compilation required on his part.
Here AI was not a replacement for test thinking, but a way to collapse the distance between “I know what I need” and “I have a tool that does it.”
#2 The pseudonymization pipeline
A few weeks later, a different team needed to move test data from production to a QA environment. The data contained personal information. The process of extracting it, pseudonymizing it, and loading it into the right place had been manual, slow, and error-prone, which meant it happened rarely, which in turn meant the QA environment was always stale.
I spent thirty minutes with the developer responsible for the system, sketching what the solution should look like. A function app that reads from production, pseudonymizes on the fly, and writes to QA via Service Bus. He said it sounded good. The next day I had a working version.
We hit deployment problems: access rights, GitHub repository permissions, Azure configuration. Another design session, thirty minutes, two people. We split the architecture: one function app for pseudonymization, separate loader apps per data type. The day after that, the new version was ready. The developer’s team took it from there.
The total elapsed time from “we have a problem” to “we have a deployable solution” was under a week. The total time I spent in meetings was under two hours. The rest was building, with Claude doing the mechanical coding and me doing the architecture, the domain reasoning, and the testing of the tool itself.
#3 The certificate debugging case
Not every AI-assisted win is a big build. Sometimes it’s a two-hour debugging session that would otherwise have taken two days.
A project had been blocked for months on a certificate and API key problem with an Altinn integration. The symptoms were clear: authentication failures. But then it turned out that the root cause was buried in how enterprise certificates were being encoded and passed. I built a small Python diagnostic tool with Claude to inspect the certificate chain, decode the key material, and compare it against what the API endpoint expected. Within 48 hours the blocker was gone.
That kind of quick, targeted tooling is where I find AI most consistently useful for clients. Immediate recognizable value. Not grand automation frameworks, but rather small, sharp instruments built for a specific problem, used once or twice, and discarded. The cost of building them is low enough that it’s almost always worth it, or at least try.
The validation burden
But there’s a cost that doesn’t show up in the time-to-build numbers, and it’s worth being honest about it: When AI generates code, someone has to validate it. Not just run it and see if it works, but actually read it, understand it, and confirm that it does what it’s supposed to do under all the conditions that matter.
That validation burden is real, and it grows with the complexity of what’s being built. I’ve started calling this “comprehension debt”. When you build something with AI assistance, you can end up with working code that nobody fully understands.
That’s fine for a throwaway diagnostic script. It’s not fine for a pseudonymization pipeline that handles personal data in production.
The more consequential the tool, the more important it is to slow down and actually understand what you’ve built. And crucially, treat it as any other piece of software and add appropriate test coverage and AI readable documentation, so the AI agent doesn’t introduce failures later on.
This is not a new problem. It’s the same problem that exists with any code written faster than it’s understood. AI accelerates the writing. It does not accelerate the understanding. That asymmetry is something every tester using these tools needs to keep in mind.
From specification to code: What changes, and what doesn’t
In the article LLMs Are Just Another Compiler. And That’s a Good Thing!, Erik Arisholm has written about LLMs as compilers, as tools that translate a specification from one notation into another. The quality of the output is bounded by the quality of the input. That framing has stuck with me, because it matches exactly what I experience in practice.
When I use Claude to build a test tool, I’m not describing vague intentions and hoping for the best. I’m specifying precisely: what data shape goes in, what transformation happens i.e pseudonymization, what comes out, what the error cases are. The better I specify, the better the result. The gaps I leave, Claude fills silently, and those silent fills are where the bugs live.
This, again, is not a new problem. It’s the same problem testers have always had with requirements. AI makes it faster to get from specification to running code. It does not make it easier to write a good specification. If anything, the speed makes the specification problem more visible, because you get to the wrong answer faster.
What hasn’t changed: you still need to understand the system. You still need to know what a good test looks like. You still need to be able to read the output and judge whether it’s actually correct, or just plausible-looking. That judgment is the job. AI handles the mechanical translation. The judgment is still yours.
What this means for testing as a discipline
In Testing as a Discipline, we make the case that testing is a discipline, not an activity. Independent verification catches things that developer self-review structurally cannot. Sustainable test automation requires architectural thinking, not just scripting.
AI doesn’t change any of that. What it changes is where the skilled tester spends their time. The mechanical work of writing boilerplate, wiring up data transformations, scaffolding test harnesses and running tests in parallel, that part is faster now. That’s good.
It means more time for the work that actually requires judgment: designing coverage, reasoning about integration failure modes, deciding what to automate and at which level, and validating that the tools you’ve built actually do what you think they do.
In my experience, the testers who get the most out of AI tools are the ones who already had strong fundamental understanding. They know what they’re specifying. They know what good output looks like. They know when to trust the result and when to dig deeper. The tools amplify what’s already there. They don’t substitute for it.
Where this is going
I’m not going to predict where AI tooling will be in two years. What I can say is that the pattern I’ve described: tester as architect and validator, AI as fast mechanical executor, that feels durable. The specification problem isn’t going away. The need for independent verification isn’t going away. The judgment required to design good tests isn’t going away. If you are making more tools and solving more complex problems, the ability to verify and validate becomes more valuable, not less.
What is going away is the excuse that there wasn’t time to build the right tooling. The question now is whether you have the skills to use the time well.
Back to blog