Test Intelligence

Updated 1 month ago by Manish Jaiswal

This topic describes Harness Test Intelligence (TI). TI dramatically improves test time by running only those tests required to confirm the quality of the code changes which triggered the build.

In this topic:

Before You Begin

Before learning about Test Intelligence, you should have an understanding of the following:

Visual Summary

The following video walks you through setting up Test Intelligence in Harness CIE stage. The TI section starts after 11 min. mark in the video.

Overview

Testing is an important part of Continuous Integration (CI). Testing safeguards the quality of your product before shipping. But testing can also take a lot of time because a test cycle involves multiple tests. Often, the tests run are irrelevant to the code changes that triggered the build.

Harness TI helps your test cycle move faster without compromising on quality or test coverage. 

At runtime, TI selects only those tests you need to run. Instead of running all your tests (which can take hours), TI selects a subset of tests and skips the rest. 

You'll have full visibility on which tests were selected and why. TI doesn’t require you to update your source code or build and test process. TI is instrumented as the bytecode on the fly.

How Does Test Intelligence Work?

Test Intelligence (TI) does the following:

  • Prioritizes and runs only important tests. Running all unit and functional tests every time the code changes is expensive and time-consuming.
  • Identifies negative trends and provides actionable insights to improve quality. 
  • Identifies gaps in the test plan, and highlights the changed code.

What Will Test Intelligence Do?

Test Intelligence builds software faster than a full build-and-test run by doing the following:

  1. Test Selection: TI runs only the tests required to confirm the quality of the code changes that triggered the build. TI also includes any changed tests.
    If the CI Pipeline is using code on the release branch, it runs only those tests that cover code changes since the last release build.
    To ensure full accuracy, TI uses the call graph of the instrumented source code. 
  2. Ranking of Selected Tests: TI runs tests in an optimal order. Any failed tests from the previous build, flaky tests in ascending order by code coverage or test execution time, or other metrics (TBD). This ranking increases the rate of fault detection (FDR %).

What are the Test Intelligence Components?

Test Intelligence Service

Test Intelligence (TI) runs as a service, agnostic to the CIE solution. 

A TI service manages the data about repos, git-commit graphs, test results, and call graphs. 

During the selection phase, a TI service uses the list of added/modified files with the call graph to identify which tests to run. Next, TI uses ML modeling to rank the tests. The higher the rank, the higher the chances of failure. This helps TI get fail signals faster. The test ranking is provided to you in the TI interface.

The TI service can also receive real-time webhook notifications from Git for any commit or merge. The TI service pulls the git commit-graph and other metadata from Git for test selection and ordering. When the TI test runner agent sends a call graph generated from a PR, the TI service keeps that data in a staging area in case the PR doesn’t get merged to the master. Once the TI receives the merge notification from Git, it updates and inserts the partial call graph with the master call graph.

Test Runner Agent

The Test Runner Agent runs in the build farm. It's responsible for communicating with the TI service. Whenever a test step is about to execute, the Test Runner Agent communicates with TI service providing the build number, commit-id, and other details. The TI service returns the ordered selected tests to run. The Test Runner Agent runs the tests with the instrumentation ON. After all the test completion, the agent parses the test results and uploads the results along with the newly generated call graph.

TI and the Test Step Packaging

The Test Step is similar to the Run Step, but it accepts additional information such as the programming language of the source code being tested, build tools (Maven, Bazel, etc), and other parameters. 

TI identifies the programming language and uses the Test step to run the selected tests in the Test step container. The Test step parses test results and returns the results to the TI service.

Viewing Tests Selected by TI

Here's an example of a specific PR. You can see the number of Selected Tests and Time Saved.  On the right, you can see a breakdown of the selection. 

We also provide full visualization of the test graph. It shows the reason behind every test selection. Click on any test (the purple node), and you can see all the classes and methods covered by this test. These are the changed classes and methods (denoted by the blue nodes), which led to the selection of that test.

What Can You Achieve With Test Intelligence?

Here’s a summary of what we achieved running Test Intelligence on our biggest BE repositories, Portal.

We not only tried Test Intelligence on our biggest BE repository and few selected customers, but also on some common open-source projects. We've collated our data and it looks very promising. It's amazing how much you can save on developers' time and money.

Project Name

Avg Test Execution Time without TI

Avg Test Execution time 

with TI

Portal (Harness)

43 mins

32 mins

Incubator Pinot

338 mins

228 mins

Hudi

58 mins

43 mins

RocketMQ

4.6 mins

3.1 mins

Spring Cloud Alibaba

0.744 mins

0.59 mins

Incubator Shenyu

1.16 min

0.4 min

Sentinel

1.90 min

1 min

What Is Our Learning Method?

We employ supervised Machine Learning as we add more knowledge on the outcome of historical tests. Our algorithm learns from the labeled data. We train an XGBoost  model, using features from historical tests. Some of the features used to train the model are:

  • Code coverage of tests
  • Time of execution of tests
  • How often did a test fail when the same files were modified
  • How often were the source files modified together with the test files

ETL

As the ML models become stale, they drift and the performance degrades. However, we keep the ML models as fresh as needed by running periodic ETL pipelines as depicted in the following diagram.

Try It Yourself

Interested in trying Test Intelligence yourself, wait no more! It's available as part of the CI Enterprise’s free trial — Sign up and give it a spin! 

See Also

Set up Test Intelligence


Please Provide Feedback