Allure Report Hands-on Guide

Allure Report is an open-source multi-language test reporting tool. It builds a detailed representation of what has been tested and extracts as much information as possible from everyday test execution.

In this guide, we’ll take a journey through the main steps to creating your first Allure report and discover all the fancy features that it brings to routine automated testing reports.

Since Allure Report has various integrations with various testing frameworks on different programming languages, there is a chance that some steps will vary for each reader, so feel free to jump into the official documentation page for details.

Installing Allure Report

As always, the first step is to install the Allure library. The exact steps vary depending on your OS:

Linux

For debian-based repositories, a PPA is provided:

sudo apt-add-repository ppa:qameta/allure
sudo apt-get update
sudo apt-get install allure

Mac OS X

For Maс OS, automated installation is available via Homebrew

brew install allure

Windows

For Windows, Allure is available from the Scoop command-line installer.

To install Allure, download and install Scoop, and then execute in the Powershell:

scoop install allure

Manual installation

Download the latest version as a zip archive from GitHub releases.
Unpack the archive to the allure command-line directory.
Navigate to the bin directory.
Use allure.bat for Windows or allure for Unix platforms.
Add allure to system PATH.

Allure CLI requires Java Runtime Environment to be installed.

Plugging Allure into the code

The next step is to provide all the necessary dependencies within the configuration file so that the build tool can use Allure.

Each framework and build tool has its own configuration settings, so the best way to get a clue on adding dependencies is to look for an example at the documentation page or get an example at GitHub.

Good news, everyone! Qameta Software, the Allure Report maintainer, is developing a GUI-based template projects generator. The templates will be standalone examples with a number of mock tests and all the infrastructure preset! Stay tuned and subscribe to our blog to get your hands on the tool among the first.

Adding annotations to the Allure Report

After plugging the Allure Report into the codebase, we can run it. However, the report will be empty.

In order to provide all the necessary data to Allure, we need to annotate the tests. There are several types of annotations:

Descriptive annotations that provide as much information as possible about the test case and its context
Step annotation, the one that allows Allure to build nice and detailed test scenarios
Parameterizing annotations that allow you to specify

Allure Report Descriptive annotations

@Epic, @Feature, @Story

A set of annotations designed to make test-case tree grouping more flexible and informative. The annotations follow the Agile approach for task definition. These annotations may be implemented on the class or on the method level.

Epic defines the highest-level task that will be decomposed into features. Features will group specific stories, providing an easily-readable structure.

As a story is the lowest part of the epic-feature-story hierarchy, the class-level story adds data to all methods of the class.

@Description

An annotation that provides a detailed description of a test method/class to be displayed in the Allure Report.

@Owner

A simple annotation to highlight the person behind the exact test case, so that everyone knows whom to ask for a fix in case of a broken/failed test. Quite useful for large teams.

@Severity

In Allure, any @Test can be defined with a @Severity annotation with any of these values like BLOCKER, CRITICAL, NORMAL, MINOR, TRIVIAL.

The severity level will be displayed in the Report, so the tester understands the severity level of a test if Failed.

Sample Tests

You may find examples of usage for all these annotations in Java code, annotations for any other technology will look quite similar:

public class AllureExampleTest {

    @Test
		@Epic("Sign In flow")
		@Feature("Login form")
		@Story("User enters the wrong password")
		@Owner("Nicola Tesla")
		@Severity(SeverityLevel.BLOCKER)
    @Description("Test that verifies a user cannot enter the page without logging in")
    public void annotationDescriptionTest() {
    }

    /**
     * JavaDoc description
     */
    @Test
    @Description(useJavaDoc = true)
    public void javadocDescriptionTest() {
    }
}

Allure Report Step Annotation

Detailed reporting with steps is one of the features people love about Allure Report. It is the @Step annotation that makes this feature possible by providing a human-readable description of any action within a test. Steps can be used in various testing scenarios. They can be parametrized, make checks, have nested steps, and create attachments. Each step has a name.

In order to define steps in code, each method should have a @Step annotation with a String description; otherwise, the step name is equal to the annotated method name.

Note that the mechanics of steps were revised and now they support smart fields analysis. In Allure 1, users had to specify indexes to refer to args' values that they wanted to inject into a step. Allure 2 uses a reflection-based approach that provides deep fields' extraction by their names.

In code, the Step annotation will look like this:

package io.qameta.allure.examples.junit5;

import io.qameta.allure.Allure;
import io.qameta.allure.Step;
import org.junit.jupiter.api.Test;

public class AllureStepTest {

    private static final String GLOBAL_PARAMETER = "global value";

    @Test
    public void annotatedStepTest() {
        annotatedStep("local value");
    }

    @Test
    public void lambdaStepTest() {
        final String localParameter = "parameter value";
        Allure.step(String.format("Parent lambda step with parameter [%s]", localParameter), (step) -> {
            step.parameter("parameter", localParameter);
            Allure.step(String.format("Nested lambda step with global parameter [%s]", GLOBAL_PARAMETER));
        });
    }

    @Step("Parent annotated step with parameter [{parameter}]")
    public void annotatedStep(final String parameter) {
        nestedAnnotatedStep();
    }

    @Step("Nested annotated step with global parameter [{this.GLOBAL_PARAMETER}]")
    public void nestedAnnotatedStep() {

    }

Allure Report Parameters Annotations

Attachment

The annotation allows attaching a String or Byte array to the report. This annotation is used to attach a screenshot or a failure stack trace to the result.

Link / Links

Well, that’s simple. If you need to add a link to the test, be it a reference or a hyperlink, this is the annotation you need.

It takes a number of parameters:

name: link text
url: an actual link
type: type of link
value: similar to name

Muted

An annotation that excludes a test from a report.

TmsLink

A way to link a result with a TMS object, if you use any. Allows entering just the test case id that will be added to the pre-configured (via allure.link.tms.pattern) URL. The annotation takes a String value, which is the link to the management system. For example, if our test case on TMS link is https://tms.yourcompany.com/browse/tc-12, then we can use tc-12 as the value.

Allure Report Launch

Local

A great way to get started with Allure Report is to run it locally. Local execution does not provide execution and results history, nor the trend graphs.

The best way to give Allure Report a try is to open an example with an IDE. Check all the dependencies and build the project with Gradle.

After that, we need to run the tests with the./gradlew test command. When the tests are executed, Gradle will store the results in the target directory.

Let’s take the data and build a report! With the allure serve /path/to/the/project/allure-example/build/allure-results command, we start the Allure Report instance and build a local web report which automatically opens as a page.

CI (Jenkins, TeamCity, and Bamboo)

Allure Report has several neat integrations with different CI systems. Each system setup has its specificities, so we won’t cover them in this post. Follow the Documentation page for steps to create a report with a CI system.

Allure Report Features

Parameterized tests

behaviors

Allure knows how to work with parameterized automated tests. Let's take a JUnit test as an example. First, let's create a test class with parameterized tests of the following kind:

@Layer("rest")
@Owner("baev")
@Feature("Issues")
public class IssuesRestTest {

    private static final String OWNER = "allure-framework";
    private static final String REPO = "allure2";

    private final RestSteps steps = new RestSteps();

    @TM4J("AE-T1")
    @Story("Create new issue")
    @Microservice("Billing")
    @Tags({@Tag("api"), @Tag("smoke")})

    **@ParameterizedTest(name = "Create issue via api")**

    @ValueSource(strings = {"First Note", "Second Note"})
    public void shouldCreateUserNote(String title) {
        parameter("owner", OWNER);
        parameter("repo", REPO);
        parameter("title", title);

        steps.createIssueWithTitle(OWNER, REPO, title);
        steps.shouldSeeIssueWithTitle(OWNER, REPO, title);
    }

As you can see, after execution, Allure provides the parameterized test run results as a set of tests. If any of the tests fail, Allure provides detailed information about that particular case.

Categories

Categories is one of the most time-saving features of Allure Report. It provides simple automation for fail resolution. There are two categories of defects by default:

Product defects (failed tests)
Test defects (broken tests)

Categories are fully customizable via simple json configuration. To create custom defects classification, add categories.json file to allure-results directory before report generation.

Open json template [ { "name": "Ignored tests", "matchedStatuses": ["skipped"] }, { "name": "Infrastructure problems", "matchedStatuses": ["broken", "failed"], "messageRegex": ".*bye-bye.*" }, { "name": "Outdated tests", "matchedStatuses": ["broken"], "traceRegex": ".*FileNotFoundException.*" }, { "name": "Product defects", "matchedStatuses": ["failed"] }, { "name": "Test defects", "matchedStatuses": ["broken"] } ]

The json includes the following data:

(mandatory) category name
(optional) list of suitable test statuses. Default ["failed", "broken", "passed", "skipped", "unknown"]
(optional) regex pattern to check the test error message. Default "."
(optional) regex pattern to check the stack trace. Default "."

Test result falls into the category if its status is in the list and both the error message and the stack trace match the pattern.

In the case of using allure-maven or allure-gradle plugins, categories.json file can be stored in the test resources directory.

Retries

Retires are executions of the same test cases (asignature is calculated based on test method name and parameters as well) within one test suite execution, e.g. when we are using TestNG IRetryAnalyzer or JUnit retry Rules. Not supported for local runs.

Test History

Allure 2 supports history for tests in the report. At each report generation during the build, the Allure Plugin for Jenkins will try to access the working directory of the previous build and copy the contents of allure-report\\history folder to the current report.

At the moment the history entry for the test case stores information for up to 5 previous results. Not supported for local runs.

Report Structure and Dashboards

Overview

Dashboards

The default page would be the 'Overview' page with dashboards and widgets. The page has several default widgets representing the basic characteristics of your project and test environments:

Statistics - overall report statistics.
Launches - if this report represents several test launches, statistics per launch will be shown here.
Behaviors - information on results aggregated according to stories and features.
Executors - information on test executors that were used to run the tests.
History Trend - if tests accumulate some historical data, a trend will be calculated and shown on the graph.
Environment - information on the test environment.

Home page widgets are draggable and configurable. Also, Allure supports its own plugin system, so quite different widget layouts are possible.