Test Intelligence

Updated 3 weeks ago by Manish Jaiswal

This topic describes Harness Test Intelligence (TI). TI dramatically improves test time by running only those tests required to confirm the quality of the code changes which triggered the build.

In this topic:

Before You Begin

Before learning about Test Intelligence, you should have an understanding of the following:

Visual Summary

The following video walks you through setting up Test Intelligence in Harness CIE stage. The TI section starts after 11 min. mark in the video.


Testing is an important part of Continuous Integration (CI). Testing safeguards the quality of your product before shipping. But testing can also take a lot of time because a test cycle involves multiple tests. Often, the tests run are irrelevant to the code changes that triggered the build.

Harness TI helps your test cycle move faster without compromising on quality or test coverage. 

At runtime, TI selects only those tests you need to run. Instead of running all your tests, TI selects a subset of tests and skips the rest. 

You'll have full visibility on which tests were selected and why. TI doesn’t require you to update your source code or build and test process.

How Does Test Intelligence Work?

Test Intelligence (TI) does the following:

  • Prioritizes and runs only important tests. Running all unit and functional tests every time the code changes is expensive and time-consuming.
  • Identifies negative trends and provides actionable insights to improve quality. 
  • Identifies gaps in the test plan, and highlights the changed code.

What Will Test Intelligence Do?

Test Intelligence builds software faster than a full build-and-test run by doing the following:

Test Selection: TI runs only the tests required to confirm the quality of the code changes that triggered the build. TI also includes any changed tests.

If the CI Pipeline is using code on the release branch, it runs only those tests that cover code changes since the last release build.

To ensure full accuracy, TI uses the call graph of the instrumented source code. 

What are the Test Intelligence Components?

Test Intelligence Service

Test Intelligence (TI) runs as a service, agnostic to the CIE solution. 

A TI service manages the data about repos, git-commit graphs, test results, and call graphs. 

During the selection phase, a TI service uses the list of added/modified files with the call graph to identify which tests to run. Next, TI uses ML modeling to rank the tests. The higher the rank, the higher the chances of failure. This helps TI get fail signals faster. The test ranking is provided to you in the TI interface.

The TI service can also receive real-time webhook notifications from Git for any commit or merge. The TI service pulls the git commit-graph and other metadata from Git for test selection and ordering. When the TI test runner agent sends a call graph generated from a PR, the TI service keeps that data in a staging area in case the PR doesn’t get merged to the master. Once the TI receives the merge notification from Git, it updates and inserts the partial call graph with the master call graph.

Test Runner Agent

The Test Runner Agent runs in the build farm. It's responsible for communicating with the TI service. Whenever a test step is about to execute, the Test Runner Agent communicates with TI service providing the build number, commit-id, and other details. The TI service returns the ordered selected tests to run. The Test Runner Agent runs the tests with the instrumentation ON. After all the test completion, the agent parses the test results and uploads the results along with the newly generated call graph.

TI and the Test Step Packaging

The Test Step is similar to the Run Step, but it accepts additional information such as the programming language of the source code being tested, build tools (Maven, Bazel, etc), and other parameters. 

TI identifies the programming language and uses the Test step to run the selected tests in the Test step container. The Test step parses test results and returns the results to the TI service.

How Does TI Sync with Tests?

TI is always up to date and syncs when you merge the code to the main branch.

When you perform a pull request and run a unit test, TI determines which tests should be run based on the following metrics:

  • Changed Code: TI queries Git to learn exactly which code has changed in a specific build. TI uses this data to select all the tests that are associated directly, or indirectly, with the source code changes. TI selects these tests as part of the subset of the tests run in the Pipeline. Lastly, TI skips tests that are not needed because there was no relevant code change.
  • Changed Tests: When a specific test change is identified, TI chooses it even if the code it covers hasn't changed.
  • New Tests: As soon as Harness identifies a new test, it is automatically selected. This ensures the test is running successfully and also finds correlations between the test and new/existing code.

After each test cycle, you have full visibility into which tests were selected by TI and why? You can also see the Test Graph of your Pipeline Tests page.

Viewing Tests Selected by TI

Here's an example of a specific PR. You can see the number of Selected Tests and Time Saved.  On the right, you can see a breakdown of the selection. 

We also provide full visualization of the test graph. It shows the reason behind every test selection. Click on any test (the purple node), and you can see all the classes and methods covered by this test. These are the changed classes and methods (denoted by the blue nodes), which led to the selection of that test.

What Can You Achieve With Test Intelligence?

Here’s a summary of what we achieved running Test Intelligence on our biggest BE repositories, Portal.

We not only tried Test Intelligence on our biggest BE repository and few selected customers, but also on some common open-source projects. We've collated our data and it looks very promising. It's amazing how much you can save on developers' time and money.

Project Name

Avg Test Execution Time without TI

Avg Test Execution time 

with TI

Portal (Harness)

43 mins

32 mins

Incubator Pinot

338 mins

228 mins


58 mins

43 mins


4.6 mins

3.1 mins

Spring Cloud Alibaba

0.744 mins

0.59 mins

Incubator Shenyu

1.16 min

0.4 min


1.90 min

1 min

Try It Yourself

Interested in trying Test Intelligence yourself, wait no more! It's available as part of the CI Enterprise’s free trial — Sign up and give it a spin! 

See Also

Set up Test Intelligence

Please Provide Feedback