In this article, I’ll share how we run automated regression tests for the SalaX Secure Messaging web client across multiple environments using Playwright and GitLab CI.
SalaX Secure Messaging https://www.ssh.com/products/secure-messaging is designed for organisations that require trust, control, and enterprise-grade security. As our Playwright automation coverage grows, we also need the same regression suite to run reliably across different application environments.
Currently, we run against:
-
QA —
Currentin our automation configuration -
Staging —
Latestin our automation configuration
We expect to add 2–3 more environments to the same framework soon.
The goal was simple:
One Playwright regression suite, multiple environments, without duplicating tests or GitLab CI jobs.
Overall Architecture:
The main pieces are:
Environment Configuration
The application URLs are stored as environment variables:
The tests themselves don't need to know which environment they are running against. GitLab provides the appropriate configuration.
Test Tagging
Normal regression tests use @Regression:
This test can run in both environments.
If an individual test currently applies only to Staging, we add @LatestEnv:
Similarly, a QA-only test can use:
@CurrentEnv
So the tagging strategy is:
Excluding Complete Test Folders
Sometimes the difference is not one test but an entire folder.
For example the folders are :
tests/attachments/** tests/attachments-new/**
For this we use TEST_IGNORE.
QA:
Staging:
This gives us a simple rule:
Individual test difference → Environment tag Complete folder difference → TEST_IGNORE
Playwright Configuration
The filtering logic is centralized in playwright.config.js:
The flow is straightforward:
For QA:
EXCLUDE_TAG=@LatestEnv
For Staging:
EXCLUDE_TAG=@CurrentEnv
This allows Playwright to filter environment-specific individual tests during test discovery.
GitLab parallel:matrix
The main improvement is using a GitLab matrix instead of maintaining separate regression jobs in the .gitlab-ci.yml file
From this single definition, GitLab creates two parallel executions:
This is the part I like most about the approach: the Playwright test command remains the same; only the environment configuration changes.
Regression Reporting
Each environment produces its own:
After both matrix jobs complete, the reporting job: send:daily-regression-report
collects the results.
This gives the team one report instead of separate messages for every environment.
SalaX Secure Messaging Report
The final GitLab bot message provides a quick environment-level summary.
For example:
GitLab still contains the detailed test results and artifacts when investigation is required, while SalaX Secure Messaging provides the team with the quick daily regression status.
Scaling to More Environments
Today:
Soon, we expect another 2–3 environments to be added to the same framework.
The idea is that we shouldn't need another Playwright framework or duplicated CI jobs.
We extend the matrix:
The same test suite, filtering logic and reporting flow can continue to be used.
Conclusion
The final approach is quite simple:
Today, this setup runs our regression suite against QA and Staging, with 2–3 more environments expected to join the same framework soon.
The key idea is:
One Playwright regression suite, multiple environments, parallel GitLab execution, and one consolidated SalaX Secure Messaging report.
Today, we run against QA and Staging, and the same architecture is ready to support 2–3 more environments as our automation framework grows.
Happy Automation testing Guys :)!
