Scaling Scenario Testing
Redesigned a code-heavy, autonomous-driving developer workflow spanning local and CI/CD pipelines into a structured GUI.
Impact: within one quarter of release
time to create a test, down from an expert-only process.
of feature teams migrated to the new workflow.
catches bad configs during setup, the biggest single contributor to the time saved.
gives engineers, leadership and marketing the same view of test status.
Business needs
A feature team managed about 45 scenarios and 10 tests a quarter. The new opportunity meant closer to 3,000 scenarios and 65 tests, across roughly 10 teams.
How might we turn an expert-heavy, code-and-CLI-driven testing workflow into a structured GUI that's faster to use and automatically traceable?
User needs
Test engineer's journey of creating and finding Training data. Insights gathered from usability testing with 2 of the 10 target feature teams.
| Scenarios | Map | Rules | Create a test | Stakeholder sharing | Total time | |
|---|---|---|---|---|---|---|
| Task | 1 week | 1 dayConfirm the map is current with the owning team | 0.5 dayFind the rule for the selected scenario | 3 daysLink the data and write the test report | 0 daysStakeholders get visibility into progress and results | 2 weeks |
| Pain point | A lot of code reading | — | Also a lot of code reading | Re-documented for every new combination | Leadership reads the report and a separate visualisation | — |
| After | 1 dayFound through catalogue search | Same dayValidated automatically | Same daySurfaced alongside the scenario | AutomaticAI links the data and writes the report | AutomaticStakeholders view results directly | 2 days |
Discovery
Scenario catalogue
The developer first finds the relevant scenarios and checks if they are error-free. The catalogue surfaces the same information through structured metadata, tables, hierarchy, thumbnails and search instead.
Configuration + Execution
Test definition
Validates the scenario, AI stack and configuration combination before a run, catching issues like non-simulation scenarios early. Configure test definition run: branch and AI-stack selection, a scenario table with validation warnings, one Submit action.
Traceability + Documentation
Results visualisation
A shared queue surfaces pass rate, instance counts and rule-level breakdowns the moment a run finishes, work that used to be communicated manually. Engineers, leadership and marketing now see it without asking.
Team report. Pass-rate summaries down to individual rules and scenarios, generated automatically per run.
What changed
Results collected after the redesigned workflow rolled out across feature teams.
| Metric | How it was measured | Recording window | Result |
|---|---|---|---|
| Team migration | Workflow adoption across feature teams | Within one quarter | 67% of teams migrated to the new workflow |
| Test creation time | Time to create a test, before and after | Post-launch | ~2 weeks, expert-only → ~2 days, largely from catching errors during creation instead of after |
| Visibility | Qualitative feedback, shared queue usage | Post-launch | Engineers, leadership and marketing could see test status without manual updates |
| Documentation burden | Qualitative feedback from engineers | Post-launch | Structured workflow captures the information needed for downstream documentation automatically |
| Scalability | Roadmap capacity vs. adopted workflow | Ongoing | Built to carry the shift from ~45 to ~3,000 scenarios across ~10 feature teams |
Future scope
Scenario discoverability
The catalogue's table already grew wider than ideal to fit more teams' metadata. As it grows toward thousands of scenarios, findability needs to keep pace.