Allurectl: respect your system, don't drown it with http-requests

Getting large-scale test automation right isn't easy. People are always looking to make their test runs quick and cheap; providing that takes smart infrastructure. One such piece of infrastructure is allurectl. It is a command line wrapper for the APIs of Allure Report and Allure Testops. It changes how test reporting is integrated with test runs, minimizing resource costs and maximizing stability. Let's walk through the traditional method of transmitting test reports (with webhooks) and compare it to allurectl.

Traditional reporting with webhooks

When running automated tests with a reporter, the usual approach is to have webhooks written in the language of your test framework. Those webhooks send http-requests to your reporter for every event inside a test, e. g. test start, test end, attachment created, etc.

This approach creates a host of problems. Imagine you've got a thousand tests, each with ten attachments. Every test sends an event signaling it has started, another that it has finished, and then ten more for the attachments. As a result, we've got 12 thousand http-requests for just one test run. And, of course, you might have to do multiple test runs together.

Under this approach, your system will constantly endure what amounts to a DDOS-attack. Dealing with such a volume of requests is a clear bottleneck - e.g., TestRail Cloud has a limit of 180 or 300 requests per minute (depending on subscription type). Processing 12k requests at this rate will take 66 or 40 minutes. This is not a theoretical figure: a client of ours who had a webhook-based tool before told us about a test run that executed for 10 minutes and then loaded for about an hour.

Another problem with this approach is that the requests are executed synchronously with the tests themselves, which means the test has to wait while the requests are sent and the attachments are loaded.

Finally, you might simply lose some of the test results if your connection isn't stable. This problem keeps reoccurring, and it is bad: your TMS is your source of truth for the results, and this is exactly where they won't be present.

Reporting with allurectl

Allurectl is, like Allure Report and Testops, language-agnostic. It doesn't depend on anything happening in your test framework. Since sending http-requests is resource-costly, it first does something much cheaper: results are saved on your hard drive. And then those results are analyzed live and sent in large batches - in a separate, asynchronous process. Where the previous approach might have sent several hundred requests, here you'll only send one.

There is a whole bunch of benefits to this approach. First and foremost, not having your system under heavy bombardment is always nice. Since all the work of analyzing and sending the results

1. takes much less time

and

2. happens separately from the tests,

your tests execute without being clogged up by requests. Our clients have reported 1.5 times faster test execution when converting to Testops. This isn't a standard situation, but it happens.

Now, about that analyzing part. With allurectl, it's easy to do something that's well nigh impossible with webhooks: prioritize what you're sending to the reporter. Specifically, the system first sends the test results, then the fixtures, then the attachments. This means:

1. you can monitor live how your tests are doing without waiting for the whole thing to finish uploading.

2. you can send green test results without attachments. Generally speaking, you don't need a screenshot if your test has passed, right? And if most tests have passed, you can drop most attachments and save yourself a lot of time uploading them.

And, of course, this solution is much more stable. The results of your test run are stored safely on your hard drive. Transferring them over the network is a separate task that doesn't threaten the data even if it fails. The system tries to send the results in batches; if it fails, it remembers which ones have failed and then retries once everything else has been transferred. If the retries fail, the job is failed. But even then, the data can be uploaded by hand because it's stored on the hard drive where the execution happened.

Migrating to Allure Testops

Allure Testops has an additional significant advantage over other reporting tools and TMS: it has an open-source counterpart, Allure Report, with several million users. This means that by the time you decide to invest into the commercial version, you'll probably already be using Allure Report - and allurectl is part of it. As a result, the transition will take about an hour, which is nothing compared to migrating a test base to a modern TMS.

If you'd like to know how to set up allurectl itself - take a look at the documentation page. The point I want to make here is that setting it up can be done at the Allure Report stage. Once you've made the big decision about using Allure Testops, all you'll have to do is change a few lines of code in your CI pipeline. Testops uses the same annotations for tests as Framework, so you won't have to fiddle with the test base in any way.

Migrating with Jenkins

Let's take a closer look. Here's an example Jenkins pipeline for Allure Report with pytest:

pipeline {
    agent any

    stages {
        stage('git checkout') {
            steps {
                git branch: '${BRANCH_NAME}', url: 'your-git-repo-here'
            }
        }
        stage('Run tests') {
            steps {
                  sh 'execution of your tests here'
            }
        }
        stage('build allure report') {
            steps {
                allure includeProperties: false, jdk: '',  results: [[path: 'build/allure-results']]
            }
        }
    }
}

Note that you'll need to install Java and somehow fetch the Allure Report binary. Now, here's the same thing for Testops:

pipeline {
    agent any

    stages {
        stage('git pull') {
            steps {
                                git branch: '${BRANCH_NAME}', url: 'your-git-repo-here'
            }
        }
        stage('Run tests') {
            steps {
                withAllureUpload(name: '${JOB_BASE_NAME} - #${BUILD_NUMBER}', projectId: '1', results: [[path: 'build/allure-results']], serverId: 'demo.testops.cloud',  tags: 'demo,${TESTS_BRANCH_NAME}') {
                sh 'chmod +x ./gradlew'
                sh 'execution of yout tests here'
            }
        }
    }
}

Migrating with GitLab

Let's look at the same transition in GitLab - again, with pytest. First, the Allure Report version:

stages:
  - test

variables:
  ALLURE_RESULTS: "./allure-results"
# variables to be used in tests
  TESTS_ENDPOINT: "https://pytest.never.never.never.give.up"
  TESTS_BROWSER: "firefox"
  TESTS_BRANCH: ${CI_COMMIT_REF_NAME}

test:
  image: python:3.9.1
  stage: test
  before_script:
  # install dependencies
    - pip install pytest allure-pytest
  script:
  # execute tests
    - pytest --alluredir=${ALLURE_RESULTS}
  after_script:
  # generate allure report files
    # requires allure libs to be installed, requires Java
    - allure generate ${ALLURE_RESULTS} — clean -o allure-report

Same comment here as before - you'll need Java for this to run, and you'll need to fetch the Report binary. Now, transitioning to Testops:

stages:
  - test

variables:
  # variables for allurectl
  ALLURE_LAUNCH_NAME: "${CI_PROJECT_NAME} - ${CI_COMMIT_SHORT_SHA}"
  ALLURE_LAUNCH_TAGS: "${CI_COMMIT_REF_NAME}, gitlab, demo, pytest, skip-live-doc, ignore"
  ALLURE_RESULTS: "./allure-results"
  # variables to be used in tests
  TESTS_ENDPOINT: "https://pytest.never.never.never.give.up"
  TESTS_BROWSER: "firefox"
  TESTS_BRANCH: ${CI_COMMIT_REF_NAME}

test:
  image: python:3.9.1
  stage: test

  before_script:
    # download allurectl
    - wget https://github.com/allure-framework/allurectl/releases/latest/download/allurectl_linux_amd64 -O ./allurectl && chmod +x ./allurectl
    # install dependencies
    - pip install pytest allure-pytest

  script:
    # run allurectl which runs the tests
    - ./allurectl watch -- pytest --alluredir=${ALLURE_RESULTS}

This ensures that allurectl is downloaded and used to run the tests. Changes are minimal and won't take up much of your time.

Conclusion

Allurectl is a solution we've pioneered, and it has been adopted by many others - a sure sign of approval. Its advantages are:

• faster test execution
• requests don't clog up the TMS and don't lag
• you can look at the test results live before the test run is finished uploading
• attachments for "green" tests can be skipped to lower the volume of data transferred over the network
• test runs aren't under threat from anything happening with your network
• since allurectl is part of Allure Framework, migrating to Allure Testops takes a minimal amount of time