A QA engineer walks into a bar. Orders 1 beer. Orders 0 beers. Orders 999999 beets. Orders -1 beers. Orders a lizard. Orders NULL beers. Orders asdfjkl; beers. Orders 1.5 beers. Then a customer walks in and asks where the bathroom is. The bar bursts into flames.

You probably know this joke. It's a surprisingly accurate description of our reality.

We build our bars for what we can anticipate, and then users arrive with everything else. The startup answer to this has always been: ship early, learn fast, iterate. Get real users in front of the product as soon as possible so you discover the bathroom questions before you figure out how to serve a lizard at a bar.

That's a good advice. I've followed it for many years. It surfaces problems sooner, but it doesn't make them go away. You still end up spending a lot of your energy building narrow solutions to specific complaints, adding another import handler, another configuration screen, fixing another edge case that someone found in production. Your test suite keeps expanding. The customers keep surprising it anyway.

What I find interesting, and what I've recently spent a lot of time working on, is a totally different approach to this problem. What if AI coding tools could lend a hand?

But first, a distinction worth making carefully.

We're missing the more interesting part of AI coding

There's a lot of hype right now around AI and code. Copilot, Cursor, Claude Code - tools that help developers build software faster, fix small bugs automatically, and find their way around a large codebase easier. They are great, and I use them daily.

But enough has been said on that.

What I find personally more fascinating is an entirely different flavor of the AI coding concept: what happens when the AI agent inside your product can write code at runtime, in production. And instead of helping you build another iteration of your product, help your users use it.

These are two completely different ideas that are currently getting lumped together under "AI and code". The first is a productivity story, whereas the second one is a combination of product design and architecture. It's easy to miss it, as it sits outside the normal frame of how we think about building software products. The implications are also totally different: different risks, different infrastructure, and a fundamentally higher ceiling on what your product can actually do.

The industry has a name for the first concept. I'm not sure we have a clean name for the second yes. The closes I've found is runtime code generation: the ability of an LLM, embedded inside a running application, to write and execute code on behalf of a user in response to something the developer has never anticipated. I believe it's a bigger opportunity than it sounds like, and one of the most under appreciated product decisions a team building with AI can make right now.

Why software always disappoints someone

Developers build for what they anticipate. Users arrive with everything else.

The gap between these two things always fails with a loud "bang!". It fails with an error message, a disabled button, a "this format isn't supported" modal. The user hits the wall, shrugs, and works around it. The developer hears about it, adds it to the backlog, and eventually ships another configuration screen, another import handler, another option in a dropdown that covers the known cases and none of the unknown ones. Then the cycle starts again.

What's remarkable is how normalized this is. Both sides have accepted it. Users expect software to say "sorry, can't do that". Developers expect to spent a significant portion of their time building narrow solutions to specific complaints rather than addressing the underlying problem.

We can distinguish two major failure modes:

The unanticipated problem. The developer never saw it coming. A user uploads a file in a format that wasn't in the spec. They ask a question nobody designed for. The tool has no path for this. The user simply cannot proceed.

The unreliably solvable problem. The developer saw it coming, but couldn't write logic flexible enough to handle it. Parsing files where the schema is unknown until the file arrives. Transforming data that varies enough between sources that an algorithm breaks on real inputs. The application either crashes or produces output that's questionable at best.

Both failures have the same root cause. The logic was written before the specific problem arrived.

A timeless example: importing a CSV file

Here's a classic problem you must have seen before.

Let's say your application needs users to import data, a CSV of employee records, a spreadsheet of transactions, or similar. There are two common options.

Option A: Give the user a template. Here's an Excel sheet, here are the column headers, please fill it in exactly like this. Users don't comply. They'll merge the first name and the last name into one column, rename a header, drop a column they assumed wasn't important, and also reorder things. Then they'll upload it to your application, and no surprise, it won't work because it doesn't follow the template. They'll of course call your customer support to complain about it. That's an example of the first failure mode: the developer never saw it coming.

Option B: Build a mapping tool. After upload, let the user tell you which column means what, this one's the first name, that one's the signup date. This works better, but it's a meaningful amount of engineering effort to build well. And it still breaks the moment the input data is shaped wrong - first and last name crammed together or a date format you didn't anticipate. Now the user is back to reshaping their own file by hand before they can even finalize the mapping step. That's failure mode two. The developer saw it coming, tried to solve it, but the solution still broke on real input.

And then someone in the room says: we have AI now. Why are we still doing it this way?

Can't I just push everything through an LLM and take the output?

When I first tried ChatGPT and realized it does a reasonable job working with structured data, the intuition was immediate: just push everything though the model and take the output.

That works fine, as long as you have a very generous token budget and a high tolerance for garbage in the output.

The problem is architectural. Language models reason in language and language is approximate. Ask a model to compute something precisely - transform a dataset, apply a formula, in principle produce output that has to be exactly right - and you're asking it to do something it wasn't built to do reliably. The errors aren't always obvious either. The model produces something that looks correct, reads as confident, and only turns out to be wrong much later.

So prompting your way out of this problem gets you somewhere, just not reliably to the right answer.

What still surprises me is how many people I talk to who are still thinking about it this way. The intuition is understandable - the models are impressive, and it feels like they should be able to handle this problem. But "feels like it should work" and "works reliably in production" are two very different bars.

What works is giving the model a way to compute rather than guess.

So what actually changes when the agent can write code?

Reframe the problem as a coding problem and the whole shape of it changes. You know your target schema. Python is an extremely good tool for deterministic data transformation. And if a developer had the exact input file in from of them at the build time, writing a script to transform it into your schema would be a fairly routine task.

That's the job for a coding agent running in production, able to improvise and adapt the moment the file arrives. The LLM doesn't even need to read the whole file to understand its shape, a handful of rows is enough. It knows the target schema that you've defined upfront. It writes a transformation script, runs it against the actual file, validates the output deterministically, and if something's wrong, the failure gets fed back to the agent so it can revise the script. Even a complex transformation usually takes just a couple of iterations.

What happens after you go this way? The user experience becomes close to magic - drop whatever file you have, and it just works. A date arrives in a format that's not ISO, the script adapts. A price column shows up in a currency you don't natively support, it gets converted. If something genuinely required is missing, say the file has no start date for any employee, the system can say so clearly, instead of failing silently or importing broken data. And critically, nobody on your team had to scope, build, and maintain a bespoke ETL pipeline for one large client who uses an unusual export format.

That entire category of work, quietly expensive, never quite finished, just stops happening.

The CSV problem is one instance, but it points at a bigger shift happening right now. One that enables a new paradigm of how applications work.

So what are the consequences of building this way?

Plasticity. At build time, you know the shape of your applications, but you don't always know what your users will bring to it. A coding agent running in production closes that gap, adapting to scenarios that were only partially defined when the product was built. The plasticity is still bounded, as you control the schemas, the runtime environment, and the guardrails. But within those boundaries, the application bends to the user instead of the other way around.

Code execution as a deterministic tool for the LLM. Language models can't generate certain things reliably on their own. Calculations go wrong. Complex multi-step processes are hard to reason through in language without something slipping. But code execution is deterministic by its nature, and LLMs are generally good at coding. Instead of asking the model to produce a correct answer directly, you ask it to write the code that produces the answer, then you run that code.

The feedback loop. With a feedback loop, the agent can iterate towards a correct solution rather than committing to a single attempt. It runs the code, validates the output, and if something's off, it adjusts and runs again. Boundary conditions that would have required a developer to handle explicitly can be discovered and resolved by the agent in real time, without ever surfacing to the user as an error.

This is already happening

You've probably seen this pattern in tools already. Most people see the magic but don't realize how it's done.

Low-code and vibe coding tools. Platforms like Lovable or Retool let users describe what they want and get working functionality back. The user never writes code. The platform does that for the user at runtime without exposing the underlying complexity.

Data analysis in chat interfaces. When someone uploads a spreadsheet to Claude or ChatGPT and ask what the trend it, the model isn't doing the math on its own. It writes a Python script, runs it in a sandboxed environment, and reports back what it found. That's why it gets the right answer. Most users don't even notice that any of this is happening.

Desktop and computer use agents. The desktop version of Claude is very keen to generate scripts, run shell commands, and iterate on the results. It does that when you ask it to manipulate images, convert a markdown file to a PDF or run some calculations in an Excel document. This enables a lot of use cases that would otherwise be nearly impossible to maintain by an engineering team.

The pattern is the same across all three. The agent writes code to solve a problem that wasn't expected at built time. The logic didn't exist until the moment it was needed.

This doesn't come for free

Obviously none of what I've described comes without trade-offs. Building a coding agent into an application requires some architectural changes, and it comes with a few caveats worth naming before you commit to it.

Sandboxing is a must. And agent that can write and execute code can, in principle, do things you didn't intend. In a production environment with real user data, that's a critical concern. The runtime environment needs to be fully isolated - from your host system, from data the agent shouldn't have access to, and from network resources that aren't part of the task. There's no room to get this wrong.

Observability is harder than it looks. When an agent writes logic at runtime, that logic needs to be inspectable after the fact. What did it write? What did it run? What cam back? You'll need some serious logging, just for usual debugging. And in regulated industries - healthcare, finance and anything with compliance requirements - the audit trail requirement will probably drive the design of AI coding features.

The cost profile is different. Runtime code generation adds latency. The agent needs to reason about the problem, write the code, execute it, and often iterate on the solution. For some use cases that's completely acceptable - for example for a CSV export, a few extra seconds and a few cents spent on LLM is a reasonable trade for not having to reshape your files. For others, it may not always be so simple. Think it through, it just works differently.

The benefits are real, and so is the work. Keep these things in mind when designing a system that will eventually need to hold up in production.

Where this goes from here.

At the end of all of this, the thing I keep coming back to is the user experience. The moment someone drops a file into an application and it just works - regardless of the format, regardless of what the developer anticipated 0 that's a different kind of software.

We've spent a long time building bars that catch fire when someone asks about the bathroom. This is the beginning of building ones that don't.