How to shorten builds using Testcontainers

Some people complain that they’re not doing integration tests “because they take too much time.” All right, the interaction with real dependencies sometimes can’t be made shorter, but there are surely ways to start the databases, message brokers, and so on faster if you happen to use Testcontainers for that.

In this entry, I’ll be using Java. However, since Testcontainers is (from a certain angle) a convenient way to interoperate with Docker-compatible containers using the programming language you know, this should apply to other technology stacks as well. I’m pretty sure that after grasping the idea, one should be able to translate it to Go or C#.

And quite naturally, this post will have some numbers, sorry.

How many times start the container?

Testcontainers can be perceived as an Infrastructure-as-a-Code (IaaC) tool. How many times do you usually start your databases or other dependencies of your infrastructure for your tests when doing that, for example, manually or in a Bash script? Do you do that before every @Test or rather start once and let the whole suite (or as many as you need) run? Exactly. For that reason, (re-)starting containers before individual test cases should be an exception rather than the default practice.

If you control your containers' lifecycle manually, in Java, it’s usually enough to call container.start() in a static method, usually annotated with @BeforeAll.

It might be less obvious if you’re relying on a testing framework integration, like annotating your JUnit test class with @Testcontainers and the container fields with @Container. Then instead of doing this:

1@Testcontainers
2class IntegrationTests {
3    @Container
4    final PosgreSQLContainer postgres = new ...
5
6    // tests go here
7}

make sure the annotated field is static:

1@Testcontainers
2class IntegrationTests {
3    @Container
4    static final PosgreSQLContainer postgres = new ...
5
6    // tests go here
7}

Because of that, it will be started only once.

How many CPU cores do you have?

The short answer is: many. So, there’s no need to start your containers one by one in a waterfall-like approach. Instead, you can start them in parallel. For example, if in your tests, you need two external dependencies (let’s say PostgreSQL and Kafka) starting in 20s and 30s respectively, in the flow I’ve seen most of the time, the whole @BeforeAll will take around 50 seconds because (pseudocode ahead):

1final static PosgreSQLContainer postgres = new ...
2final static KafkaContainer kafka = new ...
3
4@BeforeAll
5public void setupRealDependencies() {
6    postgres.start(); // this is blocking the execution for 20 seconds
7    kafka.start(); // and after 20s, this call is blocking for yet another 30 seconds
8}

If your CPU is capable of running multiple processes at the same time, why not utilise it properly? In a way, such an enqueued startup sequence is just a waste of your processing power (unless you’re using it for something else, but who’s scrolling Facebook during cloud CI builds?)

This hint is simple: try starting the containers ASAP and start them all at once (unless, of course, your test scenario involves chaos engineering, retrials, and such…)

If you control your containers manually, in Java something like this should do:

1static final PosgreSQLContainer postgres = new ...
2static final KafkaContainer kafka = new ...
3
4@BeforeAll
5public void setupRealDependencies() {
6    Stream.of(postgres, kafka).parallel().forEach(GenericContainer::start); // this call takes max(20s,30s)
7}

Of course, sometimes there are joys of ForkJoin Pool affecting Stream.parallel(), in such case maybe you’d like to use instead:

Startables.deepStart(postgres, kafka).join();

It does virtually the same, only sometimes it takes a few moments more to grasp it (e.g. for new team members).

If you’re relying on JUnit integration by applying @Testcontainers and @Container annotations, then (starting in version 1.18.0) Testcontainers can start the containers for you in parallel too, by using @Testcontainers(parallel = true), like this

1@Testcontainers(parallel = true)
2class IntegrationTests {
3    @Container
4    static final PosgreSQLContainer postgres = new ...
5    @Container
6    static final KafkaContainer kafka = new ...
7
8    // tests go here
9}

Charge your stuff in advance

If you’re staying home, it usually doesn’t bother you whether your laptop is charged or not because it can be charged at any time, and you can still work while charging it. However, if you’re going on a long trip, you might wish to charge your electric car in advance to avoid unnecessary delays along the way.

Something similar applies to Testcontainers too because the fuel they use is Docker images.

If you’re working on your own machine and run Testcontainers-powered tests from time to time (as you should be, by the way!), then it’s easy to forget that there’s no magic involved in starting the containers, but serious engineering. To start() a container, the library must first have the image in the local storage, so it fetches it or docker pulls if you will.

On desktop, it’s not a big deal because it usually happens only the very first time, and then the image stays, so aContainer.start() doesn’t involve pulling the image when called the second time and later on.

However, in cloud CIs, this is very often not the case. Your build is running on a freshly assigned, ephemeral worker that doesn’t keep images used in previous runs. That is unless you configure your pipeline to cache them, like the .jar files from dependencies.

In case you can’t cache them (JARs, Docker images, or both), the following pre-charging technique might be quite handy. The concept is quite simple: instead of pulling the images when containers are start()ed (directly or indirectly, that doesn’t matter), we shall pull them in advance. So that when Testcontainers requires them, they’re already loaded, and our tests are then blocked only for the period needed to start the container, not pull the image plus start the container.

The question is: when to pull the images.

If you decide to pull them in the Java code, then it might be somewhat late because if we start the containers in a static method, there might be some gains, but not huge. Sometimes even negligible, in fact.

How often do you profile or observe telemetry in your build pipeline? I mean, with a proper profiler and such, to check CPU, RAM, IO, and network, and push them to the limits? I used to do that, so let me share some lessons learned: you don’t want to choke any of these when they’re already saturated, but instead, you may do some pre-computation or pre-downloading when it doesn’t affect the other actions in your build. When your pipeline is downloading half of the internet (especially with npm), you don’t want it to download the other half right that instant.

Therefore, an optimization like:

• pull Docker images, and
• pull all the dependencies using Gradle or Maven

at the very same time

might not give much gain or worse: actually slow your build down. Unless you have a remote container environment configured, then it’s fine because it will be somebody else’s network busy downloading the 1.3GB Localstack image ;-).

Therefore, the following approach (in pseudo-shell) might not be the best in your pipeline:

1docker pull -q postgres:version &
2docker pull -q confluentinc/cp-kafka:version &
3./gradlew build

because they might start competing for the same resource: network. What Gradle or Maven usually start the build with is resolving the build’s dependencies, which results in fetching the JARs. Yes, using the same network that connects you to Docker Hub.

What we might want to do instead is:

1./gradlew dependencies
2docker pull -q postgres:version &
3docker pull -q confluentinc/cp-kafka:version &
4# or whatever containers you will need later
5./gradlew build

The first (blocking call) of ./gradlew dependencies will de-facto download all the JARs needed to compile and test the project. Then, when it exits, we can happily start downloading/pulling the images that we know will be later used in the test when its turn inside the build task takes place. Usually, the build isn’t network-heavy after fetching the dependencies, so we can use the network (and little CPU and IO) to pre-pull images, making them available locally when needed later. Therefore, aContainer.start() won’t be waiting for the network (usually Docker Hub), but will be ready to start start()ing the container immediately.

Please pay attention to details and make sure you finish all docker pull ... commands with & in Bash, otherwise, they will block the compilation (and others in the build task), and therefore there will be no performance improvements.

Recap

I was hoping to explain that some things with Testcontainers (which is IaaC) should not be done in a sequential/cascade/waterfall mode but in a parallel mode (if the underlying infrastructure permits).

That applies to starting the containers and pulling the images too.

Here is a relaxing photo of what not to do. ;-)

Picture by Martin Herfurt from Pixabay.