Karpathy said something you'll regret ignoring:

"You are still responsible for your software, just as before. You are not allowed to introduce vulnerabilities because of vibe coding. "

He said it while drawing the line between vibe coding and agentic engineering. Agents write more of the code now, but none of that takes the responsibility off you.

The assumption underneath that is that a careful enough reader catches the problem. But some failures don't show up in anything there is to read.

For instance, a common fear with a RAG agent is that it could hallucinate when a question asks something outside its corpus.

But such cases are actually well handled by any competent model now. If nothing in the retrieved context looks relevant, there's no material to build an answer on.

Instead, the majority of failures originate when the retrieved context has partial coverage.

The retrieval pipeline returns context that's topically correct but doesn't cover the full question, and the model completes the remainder from parametric knowledge.

There are no token-level labels in the output to tell what was generated using retrieved context and what came from weights.

Both are streamed the same way.

Detecting this for production-grade apps needs a metric written for it, one that's also aligned with principles of agentic engineering.

And the solution is actually implemented in the eval skill that comes with Google’s Agents CLI.

I described the concern to Claude Code in plain English. It read the agent's code, came back with a plan I approved.

It then reported that no built-in metric isolates the behaviour and wrote a custom rubric called corpus_abstention.

It assigned a single categorical verdict per case rather than aggregating everything into one score, since the built-in raters regenerate their rubrics each run and leave no stable number to trend.

→ GROUNDED_ANSWER
→ CORRECT_ABSTENTION
→ UNGROUNDED_ANSWER (answered entirely from outside knowledge)
→ MIXED_LEAKAGE (grounded, but slips in one unsupported claim)
→ WRONG_ABSTENTION (refused something the docs actually covered)

After this, it automatically generated 33 scenarios partitioned by where the failure could occur, like:

- in-corpus
- off-domain
- out-of-corpus but plausibly answerable
- boundary cases where the topic is covered, but a specific detail isn't.

The baseline score was 19 of 33.
- Off-domain passed 3 of 3, as expected.
- But 6 of 15 in-corpus cases retrieved the right document, cited it correctly, answered accurately, and added a claim the source never made.

The root cause was one line in the agent's instruction: "If you already know the answer to a simple question and no document lookup is needed, you may respond directly without citations."

The eval skill helped flag this, and then Claude removed it and forced retrieval on every question.

This took the suite to 30 of 33, and ungrounded answers went from 6 to 0.

The full recording of my run is below, and I worked with the Google Cloud team on this.

Agents CLI GitHub repo → https://github.com/google/agents-cli?utm_source=fnf&utm_medium=x&utm_campaign=google-cloud-july2&utm_term=avi-chawla&utm_content=agents-cli-github

(don't forget to star 🌟)

I wrote up the full build covering all six steps from install to enterprise registration.

It includes the eval scorecard, the instruction loophole the eval caught before deployment, and what the deployment process actually looks like end-to-end.

Read it below.