Multi-Environment Playwright Regression Testing with GitLab CI

In this article, I’ll share how we are scaling automated regression testing for our product SalaX Secure Messaging web client across multiple application environments using Playwright and GitLab CI.

SalaX Secure Messaging is an enterprise secure messaging solution designed for organisations that require trust, control, and enterprise-grade security. As our Playwright test coverage continues to grow, running the same regression suite reliably across multiple environments becomes increasingly important.

To support this, we have built a scalable testing architecture where GitLab CI manages parallel environment execution, Playwright handles environment-specific test selection and execution, and the results are consolidated into a single regression report delivered to SalaX Secure Messaging.

As Playwright automation coverage grows, an important challenge is making sure the same regression suite can run reliably across different application environments.

Instead of maintaining separate test suites or duplicating CI jobs for every environment, we wanted an approach where the environment-specific differences are handled through configuration.

The approach uses Playwright environment configuration, test tags, folder exclusions, GitLab CI's parallel:matrix, and consolidated regression reporting.

The goal was simple:

One Playwright regression suite, multiple environments, without duplicating tests or GitLab CI jobs.

Overall Architecture:

The diagram below shows the complete regression flow—from the scheduled GitLab pipeline and parallel environment execution to Playwright test filtering, test execution, result collection, and finally the consolidated regression report sent to SalaX Secure Messaging.

The main pieces are:

• Environment configuration

• Test tagging

• Folder-level exclusions

• Playwright filtering

• GitLab parallel:matrix

• Consolidated regression reporting

---

Environment Configuration

Application URLs are stored as environment variables.

For example:

Instead of hardcoding an environment URL in the Playwright configuration, we provide the name of the environment variable that should be used:

URL_VAR=ENV_A_BASE_URL

Playwright then resolves the actual URL dynamically:









The tests themselves don't need to know which environment they are running against.
GitLab provides the appropriate configuration for each execution.
The same approach can also be used locally through .env file.
For example:
ENV_A_BASE_URL=https://environment-a.example.com
ENV_B_BASE_URL=https://environment-b.example.com

URL_VAR=ENV_A_BASE_URL
Switching the local environment only requires changing URL_VAR:
URL_VAR=ENV_B_BASE_URL
No test code needs to change.

---

Test Tagging

Normal regression tests use @Regression:

These tests can run across all applicable environments.
Sometimes, however, an individual test is specific to one environment or is not yet supported by another.
For example:

Similarly, another test could use:

So, the tagging strategy is:

This works well when the difference between environments affects individual test cases.

---

Excluding Complete Test Folders

Sometimes the difference is not one test but an entire folder.

For example the folders are :

In this situation, tagging every individual test would add unnecessary maintenance.
Instead, we use TEST_IGNORE.

For one environment:

For another:

This gives us a simple rule:

This keeps environment-specific differences centralized instead of adding conditional logic throughout the test suite.

---

Playwright Configuration

The filtering logic is centralized in playwright.config.js:

The folder exclusion is loaded from TEST_IGNORE:

Environment-specific individual tests are handled through EXCLUDE_TAG:

The values are then passed into the Playwright configuration:

The flow is straightforward:

and:

For example, one environment can configure:

while another can use:

This allows Playwright to filter environment-specific tests during test discovery and keeps environment-specific differences centralized instead of adding conditional logic throughout the test suite.

---

Why Use Both Tags and Folder Exclusions?

Both mechanisms solve slightly different problems.

Use environment tags when only an individual test differs:

Use TEST_IGNORE when an entire folder containing several tests belongs to another environment:

The resulting rule is simple:

This avoids adding unnecessary environment checks inside the test code.

---

GitLab parallel:matrix

The main CI improvement is using a GitLab matrix instead of maintaining separate regression jobs for every environment in the .gitlab-ci.yml file.

A simplified example looks like this:

GitLab CI parallel matrix execution: A single regression job definition creates 
parallel Playwright executions for multiple environments, each using its own 
environment configuration. The results are then collected by the reporting job.


From this single definition, GitLab creates multiple parallel executions.

Basically, like this:

The important part is that all environment jobs execute the same regression command:

Only the environment configuration changes.

---

How the Matrix Works

Each matrix entry provides the variables required by Playwright.

For example:

In the Yaml file, 

Another environment can provide a different configuration:

Playwright receives those values and applies them during test discovery and execution.

The overall flow becomes:

This is one of the biggest benefits of the approach.

• The Playwright command stays the same.
• The test suite stays the same.
• Only the configuration changes.

---

Regression Reporting

Each environment produces its own regression results.

Each matrix execution can also produce environment metadata:

After all regression matrix jobs complete, a reporting job collects the artifacts.

The reporting job then creates one consolidated regression summary.This gives the team one report instead of separate messages for every environment.

---

Consolidated Regression Report

Instead of sending separate reports for every environment, the results are combined into one consolidated regression summary and delivered to the SalaX Secure Messaging team.

This provides a quick environment-level overview without requiring everyone to open the GitLab pipeline immediately.

For example:

The consolidated report gives the team a quick overview of:

• Total, passed, and failed tests for each environment
• Environment-specific test exclusions
• Excluded test folders
• Overall regression status

The SalaX Secure Messaging report provides a quick daily regression summary, while GitLab retains the detailed Playwright results and artifacts required for further investigation and debugging.

---

Scaling to More Environments

The architecture is not tied to a fixed number of environments.

When another environment needs to join the regression suite, we simply add another matrix entry.

For example:

In your Yaml file,

So, what this gives us is:

• The existing Playwright framework does not need to be duplicated.
• The existing regression command does not change.
• The reporting flow remains the same.
• The new environment simply becomes another configuration in the matrix.

---

Keeping Environment Logic Out of Tests

A key principle of this approach is: Tests should know as little as possible about the environment in which they run.

Instead of adding environment-specific conditions inside the tests, we keep those differences in the configuration.

This keeps the Playwright tests simple, reusable, and easier to maintain as more environments are added.

---

Adding a New Environment

The process is straightforward:

This makes the architecture easier to maintain as more environments are introduced.

---

Conclusion

The final approach is quite simple:

• One Playwright regression suite. 
• Multiple environment configurations.
• Environment-specific filtering only where required.
• Parallel execution through GitLab CI.
• One consolidated regression report.

The complete regression flow brings these pieces together, from scheduled GitLab execution and environment-specific configuration to Playwright testing and consolidated reporting.

The diagram shows how a single scheduled pipeline can execute the same regression suite across multiple environments in parallel, while keeping environment differences in configuration and combining the results into one report.

The main configuration rules are:

The key idea is:

One Playwright regression suite, multiple environments, parallel GitLab execution, and one consolidated regression report. 

By keeping environment differences in configuration instead of duplicating tests and CI jobs, the automation framework becomes easier to maintain and scale as additional environments are introduced.

Happy Automation testing Guys :) 🚀!