Optimizing CI Build Times

Updated 2 weeks ago by Doug Bothwell

You can use the following methods to speed up your CI builds.

Test Intelligence

Testing is an important part of Continuous Integration. Testing safeguards the quality of your product before shipping. But testing can also take a lot of time because a test cycle involves multiple tests. Often, the tests run are irrelevant to the code changes that triggered the build.

Test Intelligence dramatically improves test times by running only the tests required to confirm the quality of the code changes that triggered the CI Pipeline. 

Optimize Docker Images to Reduce Build Times

The following practices can reduce your build times significantly: 

  • Pre-build images that include all required dependencies. If most of the build time is spent downloading dependencies, you sh ould pre-build an image with all required dependencies in a separate pipeline. Set up a periodic pipeline that builds the image with all the latest dependencies and pushes it to the registry. This image will be used by all the build pipelines.
    Pre-building images with all required dependencies is more efficient than downloading them to a baseline image as part of the Build setup. This is especially true if you update your images often to ensure that they include all the latest updates. 
  • Exclude unnecessary files and packages from your images. In addition to reducing build times, this makes the resulting images smaller, simpler, and more portable. You can use dockerignore files to exclude unnecessary files and folders from your images. 
  • Sort multi-line arguments in your Dockerfile alphabetically. This makes it easier to update and avoid duplicate packages. 

For more best practices, see Best Practices for Writing Dockerfiles in the Docker documentation.

Enable Docker Layer Caching in Build Steps

Remote Docker Layer Caching can dramatically improve build times by sharing layers across Pipelines, Stages, and Steps. You can set up Docker Layer Caching in the following Build Steps:

Optimize your Build Tests

The following practices can reduce your testing and resulting build times:

  • Use mock services in your unit tests, rather than setting up and connecting to third-party services. Mockups can often test your code as well as fully-running services but with fewer resources.
  • Avoid integration tests in your build pipelines when possible. You might want to move these to a separate Pipeline.
  • Look for obsolete tests that you can delete.
  • Look for unnecessary sleep statements in your unit test code.
  • Order your tests so that the tests most likely to fail come first.

Cache and Reuse the Data for Your Fetch Operations

Caching and reusing can be useful for data that your builds need to fetch, but that you cannot include in an optimized image as described above. Caching ensures faster job execution by reusing the expensive fetch operation data from previous jobs. See the following for end-to-end-workflow descriptions:  

Increase Step Resources 

If you still find that your builds are taking too long, check your infrastructure monitoring tools for potential bottlenecks during the time windows when your builds are running. Increasing memory or CPU capacity in your Build Steps might help speed up your builds. 

Please Provide Feedback