"We have to keep the AI on the leash. I'm still the bottleneck. I have to make sure this thing isn't introducing bugs and that there's no security issues."
He said it at YC talk last year, when the worry was reliability. The models hallucinated and made mistakes no human would, so the leash implied keeping yourself in the loop and checking the output before trusting it.
The models are far better now, and the line still holds, for a reason he was not focused on back then.
Even a model that writes flawless code today still has no idea who is allowed to run it.
Correctness and authorization are different problems, and only correctness improves as the model improves.
A perfect agent still hands a tool where anyone can do anything, because permission was never part of the task.
I actually tested this in practice with Claude Code.
I asked it to build a small internal tool with a button that issues account credits. It worked first try, and running it locally, the credit applied the instant I clicked.
Nothing decided who was allowed to click it. The agent wrote the right logic and displayed a success notification.
It never checked whether the caller had the right, whether it should pause for a human, or whether anything was logged.
And this is not a bug a smarter model can outgrow because the leash was never in the code.
Identity, permissions, and audit live in the system that runs the app, not in what the agent generates.
To solve this, I took the exact same bundle and hosted it on @retool.
The credit write that fired silently on my laptop now stopped at an approval gate, resolved to a real identity through SSO, and landed in an audit log.
I wrote none of it.
The app inherited the entire boundary the moment it was deployed, and the video shows the before and after.
I also wrote a detailed breakdown of the whole thing in my recent article, and I worked with the team to put this together.
It walks through the build, the exact moment the credit write went through on my laptop with nobody checking, and then what changed when the same app ran on Retool.
It also covers why this is a property of the runtime and not something a better model fixes, which is why devs typically miss this.
the loop itself is six lines, and nobody competes on it. every serious agent framework lands on the same tiny while-loop. model reads context, calls a tool, you feed the result back, repeat until it stops asking.
so if that part is solved, what is everyone actually engineering?
the answer is everything around the model. Boris Cherny, who built Claude Code, put it plainly. he doesn't prompt Claude anymore, he writes loops and lets them run.
that shift has a name now, and it rests on four pillars that are harder than the six lines make them look. these are the parts that actually break:
→ knowing when to stop. a terminal message ends the turn, not the task. an agent will write failing code, glance around, and declare victory. "done" has to mean the tests pass, not the agent feeling good about its work.
→ keeping the context clean. long loops rot from the inside as old outputs and dead ends pile up. a worse context produces a worse decision, which adds more noise, and the agent gets dumber the longer it runs. you fight it by treating context as a budget, not a bucket.
→ tools the agent can actually use. pile on a hundred tools and it loses track of which one to reach for. writes have to be safe to repeat, because loops retry, and a retried "create customer" call leaves you with duplicate records.
→ something that can say no. left alone, an agent agrees with itself. the fix is to separate the maker from the checker so the worker never grades its own homework.
put those four together and your job changes. you stop steering the agent move by move and start designing the system that steers it.
Karpathy runs research loops overnight that tweak a script, test it, keep what works, and throw away what doesn't, with himself nowhere in the loop. he arranges it once and hits go.
the model is becoming a commodity. the loop around it is where the real engineering lives now.
the best builders stopped asking what they should tell the agent to do. they started asking what system would do this without them.
I wrote the full breakdown. the article is quoted below.
> paste your website URL > deploy TikTok + IG accounts tailored to your niche > AI generates 1000 viral videos for your product > posts it for you too