TestRail Review: Features, Tutorial, and How To Use Selenium For Better Test Case Management

Software production companies rely on creating revolutionary and innovative software products to address a particular pain point for their customers. The software development cycle sees several code changes made over time to add or improve the system's functionality.

Testing ensures every update fits the organization's quality standard and does not conflict with the existing code bed so that the final product meets the business and user expectations.

Since testing processes involve several types of testing and create an overwhelming amount of test cases, QA and dev teams leverage test management tools like TestRail to simplify the process of creating, managing, and organizing test cases.

Learn more about TestRail's features, how to start creating test cases and test runs, and configure automation testing with custom UI scripts for Selenium to create a fully operational test execution and reporting process.

What is TestRail?

TestRail is a web-based test case management tool that enables testers, developers, and other stakeholders to construct, centralize, track, and manage their company's software testing efforts. The platform hosts project management and test result documentation capabilities.

The TestRail tool can be used for agile development and testing methodologies on an intuitive interface where you can consistently execute test cases and record results to generate meaningful and accurate reports.

You can use their features for every QA process on any modern browser platform like Chrome, Safari, Firefox, Edge, etc. The QA platform aims to help you build and scale your test workflows effectively with complete end-to-end visibility to streamline the software development process.

TestRail Features

TestRail supports all types of software testing, emphasizing exploratory, manual/script-based, black box, functional, and agile testing strategies.

Here is a closer look into TestRail's suite of features to facilitate test case management, execution, and reporting functionalities:

Test Case Management

In Testrail, users can manage their manual and automated test cases, test projects, and test suites for efficient test case organization. You can categorize test cases in tests or a hierarchical structure.

Tests can be reused in subsequent test processes by storing them in a centralized repository within organized folders. Test result data can be enhanced by adding screenshots, files, and rich text when you conduct tests for multiple versions and environments in a single location.

Milestones help you track the progress and timeline of testing processes you want to be prioritized. The feature also helps you get a comprehensive view of your testing efforts for all versions of a single software product.

TestRail lets you import test cases from any test management tool in .CSV and XML formats and share test cases repositories from different projects, releases, and sprints.

Test Planning

The platform offers several customization options, including defining the test cases and results fields, assigning user roles, planning and scheduling test runs, and configuring the UI based on the personalized requirements of your workflow.

Reporting and Analytics

You can include estimations and elapsed times for test cases in every project for accurate time tracking and to optimize your testing efforts for efficiency. The tool creates striking visualizations for test reports across all test runs.

Users can access histories, traceability, and coverage reports for requirements, tests, and defects.

Integrations

You get integrations for several development and testing tools like JIRA, Bamboo, HipChat, Bugzilla, Cucumber, Jenkins, Selenium, etc., for seamless collaboration between different platforms in a central location to reduce the hassle of navigating to different domains.

Get Started With End-to-End Test Case Management On TestRail

Before you start learning how to create and manage test cases and results, let's take the first fundamental step to manage test cases and learn how to add a project to your TestRail dashboard:

1. Navigate to your Dashboard.

2. Click "Add Project" from the center or on the right side of your screen.

3. Assign a name for your test project and add an announcement text if necessary.

4. Select one of the project types:

• Use a single repository for all cases - The recommended user project type. All test cases related to a particular project will be stored in a single location. These can be divided into sections and subsections.

• Use a single repository with baseline support - This option also gives you a single test case repository for a project. Still, it offers baseline support to manage multiple test case versions simultaneously.

Here, you create a "Master" set of test cases and then split them into "Baselines" depending on the version you're testing for larger projects that can benefit from efficient parallel testing.

• Use multiple test suites to manage cases - The multiple suites project type was the only available option in previous versions of TestRail. Users choose this option for stricter divisions between testing areas for a particular System Under Test (SUT).

5. Click on the "Access" tab.

6.Select the roles and teams authorized to view and manage the project. Access can also be given to individual users.

7. Add defects based on the testing requirements using the platform's addons and integrations.

8. Click "Add Project."

Test Cases

The Dashboard section helps you manage multiple projects from a single domain to present you with progress metrics for your testing efforts using visualizations. TestRail's project dashboard contains the following tabs:

• Overview
• Todo
• Milestones
• Test Runs & Results
• Test Cases
• Reports

With the format of TestRail's dashboard laid out, let's move on to creating, managing, and organizing test cases:

1. Open a project and select the "Test Cases" tab.

2. Click on "Add Test Case."

3. Fill in the following boxes:

• Title - Name and description of the test case.

• Section - Location of the test case in a particular project section.

• Template - Determine the fields within a test case.

• Type - The type of test being created.

• Priority - Importance of individual tests.

4. Fill in the description of test steps for QA teams and success conditions. If you choose the "Test Case (Steps)" option in "Templates," you can fill out individual boxes for subsequent steps with options for adding screenshots.

5. You can add screenshots and attachments in the top-right section.

6. Click "Save Test Case."

7. You can create Sections by clicking on "Add Section" on the right side of the "Test Cases" page based on the modules and functionality of the SUT.

8. Click on "Add Subsection" to further segment test cases within a project.

9. Access a comprehensive view of your test case progress by clicking the "Tests & Results" tab on the "Test Cases" page.

Test Runs

The "Test Run & Results" tab in your project shows the active test runs and a platform where you can rerun test cases to start multiple test runs for different software product versions.

Here's how you can create test runs:

1. Click on

2. Assign a name and the authorized users for the test run. You can optimally link your test run to a "Milestone."

3. All test cases of a project are included in the test run. Click on "Select specific test cases" to choose a particular subset.

4. Select the preferred test cases.

5. Click "Add Test Run."

6. Open a test and select "Add Result" after a test is completed.

7. Add the status of your test run.

8. Click "Assign To" to add other teams and users for further verification and validation.

9. Click "Add Result."

10.View the progress and status of test runs from the main page.

11.You can add test results for multiple runs by clicking on the checkboxes for the test runs and clicking "Add Test Result" in the center of your page.

Milestones

Milestones on TestRail help you track the timeline and progress of prioritized test goals and objectives within a testing process. You can click "Add Milestone" for a test run to fill in details like names, start and end days, etc.

Users can assign Milestones as completed to separate them from active Milestones and lock them for further changes.

How to Integrate Selenium and TestRail

Selenium is an open-source test automation tool designed for web applications to run on browsers and other environments. Integrating TestRail and Selenium provides complete visibility for your testing efforts by reporting results from both automated and manual test processes.

You can integrate several tools, test frameworks, and third-party applications using the platform's RESTful API. To learn how to integrate Selenium and TestRail and streamline your testing process further, let's start with defining a basic structure of the integration process:

1. Run Selenium UI automation.

2. Parse the Selenium results file.

3. Post test results using their proprietary API.

You can parse the results files based on adding single or multiple test cases in your Selenium automation project using the guide I have mentioned in the previous section.

Triggering Test Automation

Users can write custom UI scripts to add in test runs that trigger Selenium test automation and post the test results back into the project. The UI script sends an HTTP request to the Jenkins server to trigger an automated test and store the results in an XML format.

Follow these steps to configure TestRail automation:

1. Create test cases and assign automation types to Selenium with the corresponding automation code.

2. Use the UI script to add a custom "Start Tests" button. Navigate to Administration > Customizations > Add UI Script to add the custom script.

3. Create a test run using the guide mentioned in the previous section.

4. Configure and run UI test automation with C#.Net and Selenium.

Here's what the test automation file looks like in Visual Studio:

The test case titles should correspondingly look like this:

5. The test automation process will generate a result file in the location specified in the Jenkins agent.

To parse the automation test result file and receive test names, results, and error information for failed test cases, use the following scripts:

• Code script to parse and store test information -

// This code is to parse the result xml file and get the testcase name and its outcome, error if any        
private static List<TestResults> GetTestAutomationExecutionResult(string filePath)
{
    List<TestResults> testResults = new List<TestResults>();
    // Locate and load the result file
    XElement xelement = XElement.Load(filePath);
    
    IEnumerable<XElement> results = xelement.Elements().Where(e => e.Name.LocalName == "Results");
    
    foreach (var result in results)
    {
        IEnumerable<XElement> unitTestResults = results.Elements().Where(e => e.Name.LocalName == "UnitTestResult");
        
        foreach (var unitTestResult in unitTestResults)
        {
            if (unitTestResult.Attribute("outcome").Value == "Failed")
            {
                testResults.Add(new TestResults(unitTestResult.Attribute("testName").Value, unitTestResult.Attribute("outcome").Value, unitTestResult.Value));
            }
            else
            {
                testResults.Add(new TestResults(unitTestResult.Attribute("testName").Value, unitTestResult.Attribute("outcome").Value, ""));
            }
        }
    }

    return testResults;
}

To retrieve the test case tiles and case ids of the test cases involved in a particular test run and store them in a dictionary data structure, you can follow this example where the dictionary contains { "test name", "case id"} key/value pairs:

TestDetails(key, value) => 
{“Login_With_Valid_User”, “3281”},    
{“Login_With_Invalid_User”, “3282”},
{“Demo_Failed_Test_Case”, “3283”}

• Code script to receive test details -

// This function gets test details of particular test run
private static IDictionary<string, string> GetTestsByTestRunId(string testRunId)
{
    IDictionary<string, string> testcaseDeatilsInTestRail = new Dictionary<string, string>();

    APIClient client = new APIClient("https://xxxx.testrail.io/");
    client.User = ConfigurationManager.AppSettings["username"];
    client.Password = ConfigurationManager.AppSettings["password"];
    JArray testDetails = (JArray)client.SendGet("get_tests/" + testRunId);

    foreach (var test in testDetails)

    // Filter the which are automated using selenium. Ignore other manual test cases.
    if ((int)test["custom_automation_type"] == (int)TestTypes.Selenium)
    {
        testcaseDeatilsInTestRail.Add((string)test["title"], (string)test["case_id"]);
    }

    return testcaseDeatilsInTestRail;
}

• Code script to update test result -

// bulk-add multiple test results to TestRail
ArrayList newresult = new ArrayList();
foreach (var testResult in testResults)
{
    foreach (var test in testcaseDetailsInTestRail)
    {
         // Test Title is mapping field between UI automation ans Test Rail. We are checking if Titles are matching, then update the result back to test rail.                    
         if (testResult.TestTitle == test.Key)   
         // E.g 
         if("Login_With_Valid_Use" == "Login_With_Valid_Use")
         {
             var tempData = new Dictionary<string, object>
             {
                 { "case_id", test.Value },
                 { "status_id", GetStatusIdFromCode(testResult.TestResult) },  
                 // Get UI test automation execution result and map to Test Rail Result. E.g. Pass-> 1, Fail->5 etc.                                
                 { "comment", testResult.Error}
             };
             
             newresult.Add(tempData);
         }
    }
}

// Using TestRail's .NET binding to call the APIs
APIClient client = new APIClient("https://xxxx.testrail.io/");
client.User = ConfigurationManager.AppSettings["username"];
client.Password = ConfigurationManager.AppSettings["password"];

var data = new Dictionary<string, object>
{
    {
        "results", newresult 
    }
};


client.SendPost("add_results_for_cases/" + testRailRunId, data);

After adding the functions to parse Selenium test results files and post test results on the reporting platform, you can try the integration by running the automation project.

TestRail Cons

Despite the many intuitive features TestRail provides, there are several drawbacks to consider before choosing their services as your test case management tool. In addition to the tedious configuration process, the drawbacks include:

• Expensive for smaller organizations: The pricing model can be expensive for smaller organizations as it is based on the number of users for every premium plan. Furthermore, you have to pay additional charges for some add-ons and integrations.

• No in-built test automation: The test case management tool relies heavily on integrations with test automation tools like Selenium since it lacks built-in automation capabilities. As mentioned, configuring these integrations can be highly complex for some users.

• Steep learning curve: Although you can access several text and video resources, the variety of features and customization options can be a steep learning curve for new users.

• Defective software updates: Software updates to the platform introduce several bugs and defects, like inaccessible or disappearing test data.

• Constant Internet connection mandatory: The tool requires a constant and stable Internet connection. Being a web-based application, users working from remote areas can find operating on a platform with a limited network challenging.

Considering the drawbacks and the time users would have to invest in using TestRail's features to their fullest extent, you might be drawn to search for other test management tools available in the market.

Allure Testops is an intuitive and revolutionary tool that gives your teams the power to work with accurate test data by harnessing automation and a wide array of native integrations on a centralized platform.

Allure Testops: The Ultimate Test Management Tool

Allure Testops is a robust and comprehensive open-source test management tool that enables your QA teams and developers to streamline your automated and manual testing efforts with automatic test data enrichment, extensive integrations, defect and error categorization, and test reporting features.

You can design test cases and plans to execute manual and automated tests on a centralized platform. Allure Testops lets you manage your test runs and generate test documents in real time based on your personalized ruleset.

Features

Smart Test Cases

Allure Testops's Smart Test Cases feature removes the hassle of manually updating your test result repository by automatically updating it after every run for an accurate source of test data readily available to your QA teams for rapid insights.

Developers can create meaningful updates to your software product by working on constant feedback loops, while you can store test intent on our platform to maintain a single source of truth.

The Test Cases as Code feature lets you use an intuitive UI for better version control.

Native Integrations for Optimal Software Testing

Allure TestOps' suite of native integrations covers several software testing tools, programming languages, and test frameworks ready to be executed on your preferred CI systems.

Integrations for CI systems include Azure DevOps, Jenkins, Bamboo, GitLab, GitHub, CircleCI, TeamCity, etc., with test creds, job sync, and pipeline starting and listing capabilities.

We host integrations for Java (JUnit, TestNG), Python (PyTest, Nose), JavaScript (Jasmine, Mocha), .NET, Cucumber, Go, and PHP in programming languages and testing frameworks for effective test automation strategies.

Learn more about our integrations and view the complete list of CI systems, issue trackers, and third-party TMS (including TestRail) available on Allure Testops by clicking here.

Granular Control Over Test Runs

Allure Testops lets you start, stop, resume, and rerun build jobs from the Allure build servers. You can select test cases and filter the ones you don't require to execute in build jobs.

Using our flexible dashboard widget, you can create targets, review product status, and view your testing efforts comprehensively. You can monitor critical performance metrics at a glance with detailed visualizations and create personalized KPIs for project tracking.

Pricing

Based on your company's IT capabilities, you can choose between two premium plans for Allure Testops: Allure Testops Server and Allure Testops Cloud.

Allure Testops Server is best suited for companies with adequate IT functionalities that want to maintain their test instances on in-house servers. You can try our features risk-free with a 30-day free trial.

Allure Testops for 1 - 50 users	$30/month or $27/month (annual)
Next 51 - 100 users	$20/month or $18/month (annual)
Next 101 - 250 users	$10/month or $9/month (annual)
Others	$5/month or $4.5/month (annual)

Companies with limited IT capabilities can instead opt for Allure Testops Cloud, where we assist you with maintaining your test instances by hosting them on our cloud platform. You can try out Allure Testops Cloud's features with a 14-day free trial.

Allure Testops Cloud for 1 – 30 Users	$39/month or $35/month (annual)
Next 31 – 50 Users	$36/month or $32/month (annual)
Next 51 – 100 Users	$34/month or $30/month (annual)
Others	$30/month or $27/month (annual)

Conclusion

Software testing is an essential backbone for any organization aiming to produce high-quality products that help their customers address their pain points and acquire more revenue by winning their trust.

Test phases are conducted frequently throughout the development cycle to test the validity and functionality of the existing source code and new code changes. Running multiple tests helps cover more areas based on ensuring that the system meets the business and technical requirements.

While you can harness tools like TestRail to create, manage, and organize test cases, their glaring drawbacks and heavy reliance on integration with Selenium for test automation make it hard to recommend their services.

Allure Testops is the ultimate test management tool that automatically enriches your test data, offers greater control over your test runs, and helps you maintain a consistent automation testing strategy with native integrations on a centralized platform.

Companies with IT expertise can subscribe to Allure Testops Server after trying our features risk-free for 30 days, or choose Allure Testops Cloud if you require our assistance after you try our services with a 14-day trial.

Automate your test runs and manage result documentation better with Allure Testops today!