Mastering Playwright Reporting: A Comprehensive Guide for Playwright Testers and Developers

Software testing is an essential component of the development process that enables DevOps teams to ensure the features and enhancements they add are without defects. Your customers expect a seamless experience guaranteed through extensive testing in the development phase.

Playwright is an end-to-end web application platform rapidly gaining popularity today for Chromium, Firefox, and WebKit.

While making the most out of your browser application through testing your browser applications is ideal, most users are unaware of the features and functionalities of Playwright reporting.

Learn more about Playwright, how to install it into your system, and the various reporters you can configure to create test reports through our comprehensive guide and stay ahead of the competition.

What is Playwright?

Playwright is a powerful open-source Node.js library for automating and testing web applications on various browsers like Chrome, Firefox, Safari, and Edge. Playwright can also automate browser actions like clicks, form fill-ups, and page navigation.

Playwright is an intuitive platform supporting several programming languages such as JavaScript, TypeScript, Python, and Java. The defining feature of parallelism is that you can test for multiple browsers simultaneously without the requirement of writing individual tests.

Playwright Parallelism (and Disabling Parallelism)

You can achieve parallelism with Playwright as the platform lets you run simultaneous test process threads, with every thread running a single instance of Playwright Test File. Thus, the Playwright orchestrator runs five test files simultaneously if we have 10 test files and five threads.

Remember that test files that run in parallel are independent processes that do not communicate with each other. If one test’s performance depends on the other, parallelism would be detrimental to that particular process.

Disabling parallelism here would be the best option for you, in addition to issues including a limited number of threads, memory, or CPU performance.

To turn off parallelism in Playwright, you can set the number of workers as “one” in the test.describe.configure settings, or input the following command:

npx playwright test --workers=1

How to Run Tests on Playwright

Before we come to running tests in Playwright, let’s take a quick rundown of how to install Playwright into your system using Node.js and npm (Node Package Manager):

1.Install Node.js on your system. Download Node.js and install it if you don’t have it on your machine.

2.Install and open Visual Studio Code.

3.Open the integrated terminal to run the following command:

mkdir playwright-reporters-demo

4.Open the directory.

5.Create a package.json file with default values.

6.Install Playwright.

With Playwright installed on your machine, you can move to run tests. Since we installed Playwright with default options, it created a file named example.spec.js in the tests folder.

The example.spec.js contains two tests with 6 passed (42s) as the configuration file (playwright.config.js) has three browsers set, including Chromium, Firefox, and WebKit, to be tested twice.

After this step, open the playwright-report folder to input the following command and use Playwright reporting:

npx playwright show-report

Configuring Playwright Reporting

Playwright reporting offers extreme flexibility through its built-in test reporting mechanism to create reports in various formats based on your technical requirements and use case.

Playwright Test has built-in reporters and an option to create custom reporters. The easiest way to employ the built-in reporter by default is to pass the --reporter command line option or add it as a script in the package.json file.

npx playwright test --reporter=line

If you seek greater control over your reports programmatically, you can specify reporters in the configuration file (every example illustrating the config option will use the command line playwright.config.ts).

import { defineConfig } from '@playwright/test';
export default defineConfig({

  reporter: 'line',

});

Multiple Reports

Playwright lets you create multiple reports simultaneously by defining them on the console terminal. For instance, you can create a “list” report and “json” to receive a json file along with your test results once you specify it in the command line or configuration file.

import { defineConfig } from '@playwright/test';
export default defineConfig({
  reporter: [
    ['list'],
    ['json', { outputFile: 'test-results.json' }]
  ],
});

CI Reporters

The different kinds of reporters offered by Playwright can be generated locally and on CI. The “dot” reporter is an example that does not yield too much output to clutter the log or console by producing a single character for every successful test run. It is also the default reporter on CI.

import { defineConfig } from '@playwright/test';
export default defineConfig({
  // Concise 'dot' for CI, default 'list' when running locally
  reporter: process.env.CI ? 'dot' : 'list',
});

It will detect and choose the dot reporter for tests executed on CI. If the test runs locally, it shows a list report in the console.

Built-In Reporters

As mentioned earlier, Playwright reporting offers built-in reporters that show detailed information about failures and differ from each other based on the amount of output they create for successful test runs.

Let’s dive deeper into these built-in reporters and understand their functions:

1.List Reporter

List reporter is set as the default and prints a line for each test run. The only instance where the list reporter is not the default is with CI, where the dot reporter is the default.

The command line:

npx playwright test --reporter=list

The configuration option:

import { defineConfig } from '@playwright/test';
export default defineConfig({
  reporter: 'list',
});

An example of output in the middle of a test run looks like the following (failures get listed at the end):

npx playwright test --reporter=list
Running 124 tests using 6 workers
  1 ✓ should access error in env (438ms)
  2 ✓ handle long test names (515ms)
  3 x 1) render expected (691ms)
  4 ✓ should timeout (932ms)
  5 should repeat each:
  6 ✓ should respect enclosing .gitignore (569ms)
  7 should teardown env after timeout:
  8 should respect excluded tests:
  9 ✓ should handle env beforeEach error (638ms)
  10 should respect enclosing .gitignore:

2.Line Reporter

Line reporter is a more concise alternative to the list reporter format as it uses a single line to report the test result for the last completed test while printing errors and failures only if they occur.

Line reporter is a perfect choice for large test suites to display progress without spamming the output console with unnecessary information.

The command line:

npx playwright test --reporter=line

The configuration option:

import { defineConfig } from '@playwright/test';
export default defineConfig({
  reporter: 'line',
});

An example of a completed test run with the failure reported inline:

npx playwright test --reporter=line
Running 124 tests using 6 workers
  1) dot-reporter.spec.ts:20:1 › render expected
  ===================================================
  Error: expect(received).toBe(expected) // Object.is equality
    Expected: 1
    Received: 0
[23/124] gitignore.spec.ts - should respect nested .gitignore

3.Dot Reporter

For an even more concise format of viewing test results, dot reporter uses a single character to signify the test results. A single dot represents every passed test, and a failed test is an “F.”

The dot reporter is the default on CI and helps reduce output clutter from large test results.

The command line:

npx playwright test --reporter=dot

The configuration option:

import { defineConfig } from '@playwright/test';
export default defineConfig({
  reporter: 'dot',
});

Here is an example of the dot reporter output:

npx playwright test --reporter=dot
Running 124 tests using 6 workers
······F·············································

4.HTML Reporter

For users that require a reporting tool for their test automation frameworks, Playwright reporting offers the HTML reporter that creates a dedicated folder containing test reports that can be used as a web page.

The command line:

npx playwright test --reporter=html

The HTML report opens automatically by default when some tests fail. To change and control this behavior, you can use the open property in the Playwright config. The open property has the possible values of “always,”* “never,” and “on-failure”* (default).

The configuration option:

import { defineConfig } from '@playwright/test';
export default defineConfig({
  reporter: [['html', { open: 'never' }]],
});

5.JSON Reporter

Using the JSON reporter for Playwright reporting produces an object that contains all the information about the test run. The JSON output file path must be configured for the test report, where the result of the test run gets saved in a .json file format.

The command line:

npx playwright test –reporter=json

The configuration option:

import { defineConfig } from '@playwright/test';
export default defineConfig({
  reporter: [['json', { outputFile: 'results.json' }]],
});

JSON configuration file

Alternatively, you can choose to carry out this operation with BASH:

PLAYWRIGHT_JSON_OUTPUT_NAME=results.json npx playwright test --reporter=json

6.JUnit Reporter

The JUnit reporter creates a JUnit-style xml report. Like the JSON reporter, you must configure the XML output file path while creating this report and store the test run results in a .xml format.

If you want to write a report to an xml file and run with --reporter=junit, use the PLAYWRIGHT_JUNIT_OUTPUT_NAME environment variable.

PLAYWRIGHT_JUNIT_OUTPUT_NAME=results.xml npx playwright test --reporter=junit

The configuration option:

import { defineConfig } from '@playwright/test';
export default defineConfig({
  reporter: [['junit', { outputFile: 'results.xml' }]],
});

7.GitHub Actions Annotations

Playwright reporting provides a built-in GitHub reporter to receive automatic failure annotations for GitHub actions. While other reporters can work with GitHub, they do not provide these annotations.

The configuration option:

import { defineConfig } from '@playwright/test';
export default defineConfig({
  // 'github' for GitHub Actions CI to generate annotations, plus a concise 'dot'
  // default 'list' when running locally
  reporter: process.env.CI ? 'github' : 'list',
});

Custom Reporters

You can create custom reporters for personalized use cases by implementing a class with some of the previously mentioned reporter methods. Playwright also provides an API to help with the process.

Example of a custom reporter from Playwright’s website:

import { FullConfig, FullResult, Reporter, Suite, TestCase, TestResult } from '@playwright/test/reporter';

class MyReporter implements Reporter {
  onBegin(config: FullConfig, suite: Suite) {
    console.log(`Starting the run with ${suite.allTests().length} tests`);
  }

  onTestBegin(test: TestCase, result: TestResult) {
    console.log(`Starting test ${test.title}`);
  }

  onTestEnd(test: TestCase, result: TestResult) {
    console.log(`Finished test ${test.title}: ${result.status}`);
  }

  onEnd(result: FullResult) {
    console.log(`Finished the run: ${result.status}`);
  }
}

The configuration option:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: './my-awesome-reporter.ts',
});

Now that you have a comprehensive understanding of Playwright reporting, you are ready to run tests and compile test run results using your preferred form of reporters.

However, you will eventually encounter problems maintaining a vast database of test cases and enriching these test run data to always work with accurate information. Not to mention, viewing your test cases at a glance requires analytics and visualizations.

Allure Testops solves all these problems with our centralized platform that offers these features and more to make the most of your test case data.

Allure Testops: The Ultimate Test Management Solution

Allure Testops is an open-source software quality and test management tool that aims to seamlessly integrate into your CI/CD pipelines and streamline your test processes.

We provide your QA and DevOps teams with the tools to optimize your automated and manual testing processes through comprehensive analytics, monitoring, and native integrations on a centralized platform.

Streamline Test Processes with Native Integrations

Allure Testops provides native integrations with any CI system like Jenkins, Bamboo, GitLab, GitHub, CircleCI, TeamCity, etc., to run test processes earlier in the development cycle.

We host integrations for other programming languages and testing frameworks like Java (JUnit, TestNG), Python (PyTest, Nose), JavaScript (Jasmine, Mocha), .NET, Cucumber, Go, and PHP with job sync, test creds, and pipeline listing and starting capabilities.

Click here to view all the integrations provided by Allure Testops for CI systems, issue trackers, and third-party TMS.

Smart Test Cases

The Smart Test Cases feature is an automation solution for updating your test cases in real-time after every run to give you accurate data to work with at every step. Your DevOps teams can access a consistent repository of test data on demand.

Our Test Cases as Code feature grants granular version control through an intuitive UI to expedite your CI/CD pipelines with rapid insights on a centralized platform.

Additionally, you can create targets, track product status, and monitor test processes closely with Allure Testops dashboards. The Allure Query Language lets you create personalized KPIs for greater flexibility.

Allure Testops also hosts a community-driven test report tool called Allure Report. Your teams can create striking visualizations for your test cases and drill down into specific reports to gather more insights on success rates, duration, and flakiness on a tree-based representation.

Allure for Playwright

Playwright supports third-party integrations backed by the community, including support for the Allure reporting framework.

To start up Allure reporting, install an Allure dependency to generate reports for your projects.

npm install allure-playwright --save-dev

Install the following allure command line utility dependency in your project to open Allure reports in your browser:

npm install allure-commandline --save-dev

Using the Allure reporter, you can run Playwright with the following command line and configuration option by.

The command line:

npx playwright test --reporter=line,allure-playwright

The configuration option:

const config = {
reporter: [
['line'],
['allure-playwright']],
};


module.exports = config;

To run the configuration file we created above, use the following command line:

npx playwright test --config=playwright.allure.config.js

Successful execution of the configuration option will generate a “allure-results” folder. You must now generate an HTML report using the allure command line:

npx allure generate ./allure-results

With the successful execution of this command, a folder named “allure-report” will be generated. Allure report can also be opened by inputting the command:

npx allure open ./allure-report

Pricing

Allure Testops offers two premium plans based on your company’s technical expertise and business requirements: Allure Testops Server and Allure Testops Cloud.

Allure Testops Server provides greater control over your CI processes where you maintain your test instances after we integrate our services into your pipelines. This plan comes with a 30-day free trial to try our features risk-free.

Allure Testops for 1 - 50 users	$30/month or $27/month (annual)
Next 51 - 100 users	$20/month or $18/month (annual)
Next 101 - 250 users	$10/month or $9/month (annual)
Others	$5/month or $4.5/month (annual)

For companies with limited IT capabilities, we present Allure Testops Cloud, our cloud platform where we host and maintain your Testops instances. You can try our cloud features with a 14-day free trial.

Allure TestOps Cloud for 1 – 30 Users	$39/month or $35/month (annual)
Next 31 – 50 Users	$36/month or $32/month (annual)
Next 51 – 100 Users	$34/month or $30/month (annual)
Others	$30/month or $27/month (annual)

Conclusion

Testing your software products is crucial to ensuring you provide your end-users with a consistent customer experience while keeping your team on the same page to create quality enhancements and additions throughout the software development lifecycle.

Playwright is a groundbreaking web application testing tool, and mastering its nuances can help keep you ahead of the curve while providing unique insights into your coding efforts with reporters designed for simple to complex testing use cases.

However, you still need a platform beyond displaying and storing test report data. Your organization needs to work with accurate insights and updated information to optimize testing processes to uphold the high quality of your software products.

Allure Testops offers robust analytics tools with access to powerful testing tools through native integrations in an all-in-one platform, with options for companies with limited IT capabilities to achieve similar results with assistance from our cloud platform.

Contact us to discuss your unique test management requirements. Try out Allure Testops Server for 30 days or Allure Testops Cloud for 14 days before subscribing to our premium plans. Unleash the potential of accurate test case data with Allure Testops today!