AI SaaS (ML)

AI SaaS: Governing Quality for ML Outputs

Quality Assurance, QA Process Setup

Challenge

Model outputs were non-deterministic, so a simple pass-or-fail test did not apply. The team needed a way to tell acceptable variation from a real regression before shipping a new model version.

New versions were evaluated by people trying a handful of prompts and deciding whether the answers felt better. Different reviewers reached different conclusions, and a version that improved one type of request could quietly get worse at another. Problems often came to light only after customers noticed them.

The QA process that worked for the rest of the product didn’t fit this part of it, and there was no shared definition of what “good enough” meant for the model.

Solution

We introduced golden-set testing with agreed quality thresholds and a sandbox for each model version, so new versions are compared against known-good outputs before they reach users.

Worked with product and data science to build a golden set of representative requests with reviewed, known-good outputs, including difficult and edge cases.
Agreed measurable quality criteria and thresholds for each type of request, so the verdict no longer depends on who is reviewing.
Set up a sandbox where each model version runs against the golden set in isolation before any release decision.
Added a human review step for results close to the threshold, where judgement is still needed.
Made updating the golden set part of the routine as the product and its users change.

Results

Model updates are evaluated against a reference set instead of being judged by feel, and drops in output quality are caught before release.

Release decisions for model versions rest on criteria the whole team agreed on.
Regressions in specific request types show up in the comparison, not in customer feedback.
QA now covers the non-deterministic part of the product as well as the rest of it.