A report on the integration layer of vibe-coded software — where AI-built apps actually break, how long the failures stay invisible, and what that delay costs.
Where AI-built applications break, how long it takes anyone to notice, and why the answer to both is the same.
Generative coding tools have collapsed the cost of producing a working application. They have not changed the cost of operating one. This report examines the seam where those two facts meet: the integration boundary, where an application receives requests from services it does not control.
Across the sample described in §7, defects reported during the build phase cluster in the interface and in application logic — the parts a developer can see. Defects reported after real users arrive invert almost completely and concentrate at the integration boundary, which accounted for 48% of post-launch reports against 9% during the build. The dominant single cause is not a logic error but a shape mismatch: a handler written against the canonical payload in a provider's documentation, receiving the non-canonical payload a live account actually sends.
These failures are unusually quiet. Nothing in the visible path breaks — the payment succeeds, the provider records a delivery, the interface renders normally — so the median unaided detection time in the sample was 9.2 days, against under four minutes where the request itself was being captured. The report models the commercial consequence of that gap and concludes that the corrective intervention is not better code generation but request-level observability at the boundary: capture the real payload, read it, and replay it.
In February 2025 Andrej Karpathy described a way of working in which you “fully give in to the vibes” and accept what the model produces without reading every line.[1] The phrase stuck. By the end of that year Collins had made vibe coding its Word of the Year.[2] What began as a description of a weekend habit is now how a large share of new consumer and B2B software gets built.
The economics are genuinely different. A person with a clear idea and no professional engineering background can now put a functioning product in front of a paying customer in an afternoon. Authentication, a database, a dashboard, a checkout — each is one instruction away.
What has not changed is the second half of the job. Software does not stop at the moment it works on your machine; it starts there. And the part that most reliably goes wrong afterwards is the part nobody watches, because it does not happen in a browser and it does not happen in your editor. It happens as one server sending an HTTP request to another.
The model can write everything your application does. It cannot see anything your application receives.
This is not a criticism of generated code. Human-written handlers fail in exactly the same places for exactly the same reasons. The difference is one of exposure: a developer who wrote the handler by hand has usually read at least one real payload while writing it. A developer who asked for the handler in a sentence typically has not read one at all — and the tooling around them assumes they will never need to.
Defect reports were classified by the layer at which the failure originated, and split by phase: those raised while the application was still being built and tested by its author, and those raised after it had real users. The two distributions are close to inverses of each other.
The inversion has a mundane explanation. While you are building, you are the only user, and you exercise the application the way you imagine it will be used — through the interface, in the happy path, with data you invented. The integration boundary is barely touched, because you are not making real payments or generating real orders. Once real users arrive, that boundary carries live traffic for the first time, and it carries it in shapes nobody tested.
Ask any current model for a Stripe webhook handler and you will get good code. It will verify the signature, switch on the event type, and update your database. It is not sloppy work. It is work done against the wrong specimen.
Documentation shows a canonical payload: every field populated, the common event type, the shape that best explains the concept. Live traffic is not canonical. Optional fields are absent rather than null. Amounts arrive as integers in the smallest currency unit. Metadata you never set does not appear as an empty object — the key is simply missing, and data.object.metadata.plan throws before your logic ever runs.
| Field pattern | What a generated handler assumes | What live traffic sends | Endpoints affected |
|---|---|---|---|
metadata.* |
Present, an object with your keys | Key absent entirely when never set at checkout | 38% |
customer |
An expanded object | A bare ID string unless expansion was requested | 31% |
amount_total |
Decimal currency, e.g. 49.00 |
Integer minor units, e.g. 4900 |
24% |
type |
The two or three types in the example | The full stream, including *.updated and *.failed |
44% |
| Request body | Already parsed JSON | Raw bytes — parsing first breaks signature checks[3] | 19% |
| Delivery | Exactly once | Retried after any non-2xx, so handlers must be idempotent[3][4] | 27% |
Table 1. The divergences are individually trivial and collectively fatal. None of them is a bug in the generated code; each is an assumption that the documentation encouraged and live traffic contradicts. Source: Eventfy internal analysis of captured payloads (§7). “Endpoints affected” is the share of sampled endpoints exhibiting at least one instance. Categories overlap, so the column does not sum to 100. Behaviours marked with a citation are documented by the provider.
The pattern is consistent enough to state plainly: the generated handler is not usually wrong about what to do. It is wrong about what it is holding.
Nearly three quarters of integration failures in the sample would have been prevented by reading a single real payload before writing the handler.
A failure at the integration boundary produces no signal on any surface anyone is looking at. The customer's card is charged, so the payment provider records success. Your site never throws, so your host reports nothing. The interface renders exactly as designed, because the interface was never involved. The provider dutifully retries for days[3] against an endpoint that keeps returning the same error, and nobody reads the result.
The consequence is a detection curve unlike anything in the rest of software. Errors that surface in a browser are found in minutes. These are found when a human being writes in to ask why nothing happened.
The two curves are not measuring different levels of diligence. They are measuring whether the request existed anywhere a human could look at it. Below, the same afternoon runs twice — once with no record of what arrived, and once with the request captured.
The variable that determines detection time is not skill, seniority or how the code was written. It is whether the request was stored.
Detection time only matters because failures accumulate while it elapses. A silent handler failure is not one lost customer; it is one lost customer per occurrence, every day, until somebody notices. The model below makes that arithmetic explicit rather than rhetorical — every input is yours to set, and the formula is printed underneath.
At 4.0% silent failure and 9.2 days to detection, 1.5 of 120 monthly customers hit the broken path before anyone looks.
If the dominant cause is a shape mismatch (§3) and the dominant cost is detection latency (§4–§5), then the corrective intervention is not a better model, a stricter type system or more tests written against invented fixtures. It is a record of the request itself, kept somewhere a person — or their assistant — can read it.
In practice that decomposes into four capabilities, and they have to be present together. Any three of them still leaves you guessing.
localhost, after changing the code. Without this, verifying a fix means manufacturing another real payment, order or signup, which is why so many of these bugs are shipped hopefully rather than tested.Every capability above is retrospective. None of them prevents the first failure — they make the first failure the last one, instead of the first of ninety.
Eventfy exists to provide those four. That is the commercial interest behind this report, and it is stated here rather than implied: we sell the instrument, so read §7 and §8 before you take the numbers at face value.
Population. 312 applications whose authors self-identified as having built the majority of the codebase with an AI coding tool (Claude Code, Cursor, GitHub Copilot, Lovable, Bolt, Replit Agent, v0 or ChatGPT), and which had at least one live third-party integration in production.
Defect corpus. 1,486 defect reports classified by originating layer and by phase (pre-launch or post-launch). Classification was single-label, assigned to the layer where the failure originated rather than where it was noticed.
Boundary sub-corpus. 713 of those reports were integration-boundary failures with a recoverable first-detection timestamp; these underpin Figures 2 and 3 and both medians quoted in the abstract.
Detection timing. Time-to-detection is measured from the timestamp on the originating request to the first recorded human acknowledgement that something was wrong. The observation window closes at 30 days; failures undetected at that point are right-censored and reported as such rather than imputed.
Provenance and its limits. Figures are derived from Eventfy's own captured-traffic dataset combined with a synthetic reference corpus used to model the unaided detection curve, since by definition an unaided failure produces no telemetry to measure. They are illustrative of a pattern we observe, not the output of an independent, peer-reviewed or externally audited study, and they should not be quoted as an industry benchmark. Where a claim describes documented provider behaviour rather than our data — retry policy, signature verification, response deadlines — it is cited to the provider's own documentation in the references.
Reproducibility. If you would like the classification schema or the figure data as JSON, write to [email protected] and we will send both.
The interesting thing about vibe coding is not that the generated code is bad. In the sample it mostly was not. The interesting thing is that generation has moved the bottleneck: it used to sit in producing a working application, and it now sits in understanding one you did not write, at the exact boundary where it meets systems you do not control.
That boundary is not visible from inside your editor, it is not visible in your browser, and it is not visible to the model that wrote the handler. It becomes visible the moment the request is written down — which is a tooling decision, not a skill one, and takes about as long as pasting a URL.
You do not have a code problem. You have a visibility problem that only looks like a code problem.
The four capabilities in §6, as the product that provides them. Paste one URL into whatever is calling your app and the rest of this page stops being theoretical.
Keep working exactly as you do now. Nothing here asks you to write the handler yourself — only to look at what it receives.
Full payload, headers, status returned and every delivery attempt. Copy the JSON straight into Cursor or Claude Code and the next version of the handler is written against your data.
Send the same event again, as many times as it takes. No second payment, no second signup, no waiting for the bug to happen to somebody else.
The MCP server gives Claude Code, Cursor and Claude Desktop the failed events, the true payload shape and the ability to re-fire. Included from Pro.
17 tools covering webhook URLs, events, re-fires and Flows. Included from Pro.
Flows and the AI integration start at Pro, so that is where this page starts. There is a cheaper Individual plan for plain webhook debugging — it does not include either.
Card saved, not charged. Nothing is billed until the trial ends, and you can cancel any time. Compare every plan.
Build it with AI. Understand what it’s doing. Fix it when it breaks. Automate what happens next.