Written by 6:07 am Cloud

Implementing End-to-End testing for a Kafka Producer and Consumer that uses Spring Cloud Stream

End-to-end (E2E) testing is a critical part of any software development process. It involves testing the entire application stack, from the user interface to the backend systems, to ensure that everything works together as expected.

For Kafka-based applications, E2E testing can be a bit challenging, as it requires spinning up a Kafka cluster and deploying all of the microservices involved in the application. However, there are a few tools and techniques that can make this process easier.

One such tool is Testcontainers. Testcontainers allows you to start and stop Docker containers on demand, which is perfect for spinning up a Kafka cluster for testing purposes.

Another tool that can be helpful for E2E testing is Spring Cloud Stream. Spring Cloud Stream provides a high-level abstraction for interacting with Kafka, which makes it easier to write tests.

In this article, we will show you how to implement E2E testing for a Kafka producer and consumer that uses Spring Cloud Stream.


To follow along with this article, you will need the following:

  • Java 17
  • Maven 3.8.1
  • Docker
  • Testcontainers

Creating the E2E testing project

First, we need to create a Spring Boot application for the E2E tests. We can use Spring Initializr to create the project.

When creating the project, select the following dependencies:

  • Spring Web
  • Spring for Apache Kafka
  • Testcontainers

Creating the Kafka producer and consumer

Next, we need to create the Kafka producer and consumer that we will be testing.


public class KafkaProducer {

    private final KafkaTemplate<String, String> kafkaTemplate;

    public KafkaProducer(KafkaTemplate<String, String> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;

    public void send(String message) {
        kafkaTemplate.send("my-topic", message);


public class KafkaConsumer {

    private final ConsumerRecordFactory<String, String> consumerRecordFactory;

    public KafkaConsumer(ConsumerRecordFactory<String, String> consumerRecordFactory) {
        this.consumerRecordFactory = consumerRecordFactory;

    public List<ConsumerRecord<String, String>> consume() {
        List<ConsumerRecord<String, String>> consumerRecords = new ArrayList<>();

        ConsumerIterator<ConsumerRecord<String, String>> consumerIterator = consumerRecordFactory.iterator();

        while (consumerIterator.hasNext()) {
            ConsumerRecord<String, String> consumerRecord = consumerIterator.next();

        return consumerRecords;

Writing the E2E tests

Now that we have created the Kafka producer and consumer, we can start writing the E2E tests.

public void testKafkaProducerAndConsumer() {
    // Start a Kafka cluster using Testcontainers
    KafkaContainer kafkaContainer = new KafkaContainer();

    // Create a Kafka producer and consumer
    KafkaProducer kafkaProducer = new KafkaProducer(kafkaContainer.getKafkaTemplate());
    KafkaConsumer kafkaConsumer = new KafkaConsumer(kafkaContainer.getConsumerRecordFactory());

    // Send a message to the Kafka topic
    kafkaProducer.send("Hello, world!");

    // Consume the message from the Kafka topic
    List<ConsumerRecord<String, String>> consumerRecords = kafkaConsumer.consume();

    // Assert that the message was consumed correctly
    assertEquals(1, consumerRecords.size());
    assertEquals("Hello, world!", consumerRecords.get(0).value());

    // Stop the Kafka cluster

Running the E2E tests

To run the E2E tests, we can use Maven.

mvn test

If all of the tests pass, then we can be confident that our Kafka producer and consumer are working correctly.


In this article, we have shown you how to implement E2E testing for a Kafka producer and consumer that uses Spring Cloud Stream. We used Testcontainers to spin up a Kafka cluster for testing purposes, and we used Spring Cloud Stream to make it easier to write the tests.


Visited 138 times, 1 visit(s) today