# Replay QA: What Its 900-Bug Test Shows About AI-Built Apps

Replay's latest testing results put the focus on the evidence a coding agent needs before it can reliably fix a broken app.

By Paras Madan | 2026-11-04


Replay says its AI testing tool found more than 900 bugs across 40 apps promoted by major app builders. Its November 3 update also describes new capabilities for reproducing user sessions from partial, redacted data. The useful question is how those findings become fixes developers can verify.

## What the latest results show

In its [November 3 report](https://replay.io/blog/an-end-to-buggy-web-apps), Replay says it tested apps that had won contests or been promoted by platforms including Lovable, Replit, Base44 and Bolt.new. Astra checked and deduplicated the findings.

This is a company-run experiment on a selected sample. It does not establish the bug rate of every app built with those platforms, or how often Replay misses a problem. Replay QA itself launched months ago. This update is not its first release. Replay also acknowledges that its detection and diagnosis remain imperfect.

## How Replay QA works

The [product workflow](https://replay.io/how-it-works) starts with a web app URL. Agents explore the app, identify user journeys and run browser tests. Those runs produce deterministic recordings: the recorded execution can be replayed consistently, including the page state, network activity and JavaScript execution.

That gives the investigating agent something more concrete than a screenshot of a broken interface. It can inspect what happened during the failing run and connect visible behaviour to the code involved.

Reports include evidence, a proposed cause and a suggested fix. After a change, Replay can run the journeys again to check the original failure and look for regressions. GitHub-triggered runs and scheduled checks let this process sit alongside the team's existing development workflow.

**The distinction to keep is between suggesting a repair and verifying it.** A plausible patch still needs a test against the failure that started the investigation.

## What self-healing means in practice

Replay's earlier [October pilot](https://replay.io/blog/what-our-experimental-self-healing-software-loop-shipped-in-two-weeks) gives useful context for the latest update. That experiment connected Fullstory for production signals, Replay QA for diagnosis and verification, and Obvious for code changes. An engineer still decided whether to merge.

Over two weeks in Replay's own repository, the loop opened 152 pull requests. Engineers merged 80 verified fixes; those PRs covered 76 distinct bugs. The remaining pull requests were still open or had been closed without merging. These are different measures, so 80 merged PRs should not become a claim of 80 unique bugs fixed.

That is an internal pilot, not an independent production benchmark. It does show the shape of a useful workflow: a bug report carries evidence into the coding step, and the proposed fix returns to a verification step before someone accepts it.

## Pricing and where it fits

Replay's [pricing page](https://replay.io/pricing) lists a free plan with 25 credits per month. Monthly Starter and Pro plans are $20 and $200, with 50 and 500 credits. Annual billing displays equivalent monthly prices of $17 and $170, billed as $204 and $2,040 a year.

Credits are the product's usage allowance, not a promise that each credit represents one confirmed bug or one merged fix. Teams should evaluate the work completed within that allowance before estimating their ongoing testing cost.

For a team shipping AI-generated web applications, the practical trial is straightforward: choose a few important user journeys, inspect the evidence behind the findings and see whether the proposed repairs survive another run. Counting reports alone would reward a noisy tester.

As code generation gets cheaper, useful testing tools will be judged by how much uncertainty they remove before deployment. The valuable output is a failure someone can reproduce, followed by a change they can check.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help technical founders turn product updates and engineering work into stories developers can understand and share.

[Book a 30-minute call](https://cal.com/paras-madan/30min).