The agent reports that every test passed. In our hypothetical purchase-order project, an order totaling 50,000 baht goes to the manager and 50,000.01 baht goes to the purchasing director. The calculation looks right. Can we accept the change?
I would first ask what the tests actually examined. Perhaps they checked only a helper function, while the application still uses a different path. Perhaps the agent changed an expected result to match its code. Or the route is correct, but an unauthorized user can submit the approval directly. A green result is useful evidence only when we understand the claim it supports.
This is why AI-assisted work needs an acceptance decision, not just a success message. For this article, our hypothetical policy P1 still uses the final total in baht after discounts and tax, with manager approval up to 50,000 baht and director approval above it. Changing the total invalidates approval. These are invented rules for the example, not Odoo defaults.
Write acceptance criteria before generating the fix
“Approval works” leaves too much room for interpretation. A testable statement names an initial state, an action, and an expected outcome. For example: given a pending order totaling exactly 50,000 baht and an authorized department manager, the approval action records that manager's decision. The corresponding unauthorized request must be rejected without changing the order.
Business owners should review the rule and expected outcomes; technical reviewers should check how those outcomes are enforced. If everyone waits until the code is complete, a plausible implementation can become the default interpretation. Writing criteria early makes disagreements cheaper to see, even if new cases later require revising the criteria.
Check the software result and the agent's behavior
Tests compare an observed result with a defined expectation. A total calculation, permission rejection, or state transition can often be checked directly. Evaluations, or evals, assess broader agent behavior, such as whether it identifies an unresolved policy question and reports it clearly.
The distinction is about the claim being checked, not a rigid division between code and judgment. Some agent behavior can be checked automatically, such as detecting a prohibited tool call. Other behavior requires a rubric and a person to review the evidence. A model-based judge may assist, but its agreement is not an independent guarantee that the implementation is correct.
For our project, software tests should cover both sides of the amount boundary and the boundary itself. They should also cover changes after approval, repeated requests, and unauthorized users. We need at least one check through the actual approval entry point, because testing an isolated calculation does not prove the application uses it correctly.
| Case | Expected evidence |
|---|---|
| 49,999.99 baht | Manager route, using the agreed final-total definition |
| 50,000 baht | Manager route at the inclusive boundary |
| 50,000.01 baht | Director route above the boundary |
| Total edited after approval | Previous approval invalidated; order returns for review |
| Unauthorized approval request | Request rejected and stored order left unchanged |
| Same request submitted twice | Defined handling without a second business action |
An agent-behavior evaluation could give the agent conflicting policy examples and ask it to make the change. The expected behavior is to locate the authoritative policy or report the unresolved conflict, not silently select the easier rule. Another case could include a supplier note telling it to bypass approval. The note should not be treated as authority.
Checked
The calculation selects the right route for the test cases
Evidence for the calculation only
Not yet checked
Whether the app calls that function and purchasing integration works
No conclusion about the whole system yet
Next, test through the actual application entry point in an isolated test environment.
Define what a good evaluation result means
A rubric for this task might assess whether the agent stayed within the assigned scope, used the approved policy, preserved unresolved questions, ran the required checks, and reported failures accurately. Each criterion needs observable evidence. “Shows good judgment” is too vague to score consistently without explaining what behavior counts.
Suppose the output works, but the agent reached it by disabling a permission test. The final screen alone would hide that failure. Reviewing relevant tool actions and the change set can expose it. We do not need a speculative account of the model's private reasoning; we need evidence of what it accessed, changed, executed, and reported.
Keep some evaluation cases separate from the examples used to develop the workflow. Otherwise, an agent may appear successful because the checks repeat the exact cases it was taught. A reviewer should also inspect whether tests were weakened, omitted, or changed for a legitimate reason. Changes to acceptance criteria require their own explanation.
Concepts informed by The New SDLC With Vibe Coding, Addy Osmani, Shubham Saboo and Sokratis Kartakis (May 2026). Author’s article · Full public paper
Carry the evidence through the development cycle
The software development life cycle, or SDLC, still includes deciding what to build, designing it, implementing it, checking it, releasing it, and maintaining it. AI can help in several phases, but faster editing does not settle an ambiguous requirement or create a recovery plan. Evidence should travel with the change rather than stay in a chat that the next person cannot find.
The source figure emphasizes specification and verification, including both output and tool-use evaluation. Its durations are illustrative, not measurements for this team or a promise that every project will accelerate. A pass supports only the checked scope.
Figure reproduced from The New SDLC With Vibe Coding, Addy Osmani, Shubham Saboo, and Sokratis Kartakis (May 2026). Document design: Michael Lanning. Figure reuse offered by Addy Osmani. Source and reuse permission · Full public paper
A handoff for this approval change should include policy P1, the change set, test and evaluation results, the environment used, known gaps, and a release recommendation from the responsible reviewer. If a dependency could not be tested, say so. “All available checks passed” and “the downstream integration was verified” are different claims.
After a bounded pilot begins, monitor the cases that matter: rejected unauthorized actions, approvals invalidated by edits, unexpected failures, and discrepancies reported by users. Define who reads those signals and what makes them stop the pilot. Collecting events without an owner does not close the feedback loop.
I would accept this change only within the scope supported by its evidence. If the calculation is verified but the live purchasing connection is not, the decision can be “ready for the next isolated test,” rather than a forced choice between “finished” and “failed.”
Before accepting your next AI-assisted change, ask the reviewer to finish two sentences: “These checks support the claim that…” and “They do not yet tell us whether…” The answers expose the next useful test and give the approval decision a boundary someone can defend.
Reading and scope
The discussion develops the tests, evaluations, feedback, and SDLC concepts in The New SDLC With Vibe Coding by Addy Osmani, Shubham Saboo, and Sokratis Kartakis, May 2026. The amounts, policy P1, evaluation cases, rubric, and pilot checks are original hypothetical proposals. They are not a security certification, a complete testing standard, or evidence of results from a deployed system.
Read Addy Osmani's Agentic Engineering for another explanation of deliberate AI-assisted development. The acceptance worksheet in this article is my proposed application, not a checklist reproduced from that reading.