How to Configure Kafka in Quarkus Using application.yml

June 28, 2025

Apache Kafka and Quarkus make a powerful duo for building high-performance, event-driven microservices. If you're working with messaging in a Quarkus application, configuring Kafka properly for both local development and production is essential. The good news? With Docker, Kubernetes, and profile-specific application.yml files, it's quite manageable.
In this guide, you'll learn how to configure Kafka in Quarkus using application.yml, using Docker for your local dev environment and Kubernetes for production. We'll structure the configs around the dev and prod profiles to keep things clean and flexible.

1. Add Required Dependencies in `pom.xml`

Before anything else, make sure your Quarkus project includes the necessary dependencies:

<dependencies>
  <!-- Kafka Messaging Support -->
  <dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-smallrye-reactive-messaging-kafka</artifactId>
  </dependency>

  <!-- YAML Configuration Support -->
  <dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-config-yaml</artifactId>
  </dependency>

  <!-- Kubernetes Config Integration -->
  <dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-kubernetes-config</artifactId>
  </dependency>
</dependencies>

These allow your app to connect with Kafka and dynamically load configuration from YAML files and Kubernetes ConfigMaps/Secrets.

2. Set Up Kafka with Docker for Development

For local testing, running Kafka via Docker is the quickest route. You can use the Bitnami image:

docker network create kafka-net

docker run -d --name zookeeper --network kafka-net \
  -e ALLOW_ANONYMOUS_LOGIN=yes \
  -p 2181:2181 bitnami/zookeeper

docker run -d --name kafka --network kafka-net \
  -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 \
  -e KAFKA_CFG_LISTENERS=PLAINTEXT://:9092 \
  -e KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
  -p 9092:9092 bitnami/kafka

Kafka will be available at localhost:9092.

3. Configure `application-dev.yml`

Now create an application-dev.yml file in your src/main/resources directory:

"%dev":
  mp:
    messaging:
      incoming:
        kafka-in:
          connector: smallrye-kafka
          topic: dev-topic
          bootstrap.servers: localhost:9092
          group.id: dev-group
      outgoing:
        kafka-out:
          connector: smallrye-kafka
          topic: dev-topic
          bootstrap.servers: localhost:9092

This config enables a Kafka consumer (kafka-in) and producer (kafka-out) using your locally running Kafka broker. Make sure the topics match your use case.
Run your app using the dev profile:

./mvnw quarkus:dev -Dquarkus.profile=dev

4. Set Up Kafka Config for Production with Kubernetes

In production, Kafka will likely be managed by a cluster operator or external service. You shouldn’t hardcode hostnames or credentials. Instead, use Kubernetes ConfigMaps and Secrets.

4.1: Create a ConfigMap

Here’s an example ConfigMap to store non-sensitive Kafka configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-config
data:
  kafka.bootstrap.servers: kafka-service:9092
  kafka.topic: prod-topic

4.2: (Optional) Create a Secret

If your Kafka cluster requires credentials, create a Secret:

apiVersion: v1
kind: Secret
metadata:
  name: kafka-secret
type: Opaque
data:
  kafka.username: cHJvZHVzZXI=       # base64 for 'produser'
  kafka.password: cHJvZHBhc3M=       # base64 for 'prodpass'

Mount these in your Quarkus deployment or set them as environment variables.

5. Configure `application-prod.yml`

Now create your application-prod.yml file:

"%prod":
  mp:
    messaging:
      incoming:
        kafka-in:
          connector: smallrye-kafka
          topic: ${kafka.topic}
          bootstrap.servers: ${kafka.bootstrap.servers}
          group.id: prod-group
          username: ${kafka.username}
          password: ${kafka.password}
      outgoing:
        kafka-out:
          connector: smallrye-kafka
          topic: ${kafka.topic}
          bootstrap.servers: ${kafka.bootstrap.servers}
  quarkus:
    kubernetes-config:
      enabled: true
      config-maps: kafka-config
      secrets: kafka-secret

This tells Quarkus to fetch the Kafka config and credentials from Kubernetes at runtime.

6. Switching Between Profiles

Quarkus makes profile switching simple. You just need to set the quarkus.profile property.
For development:

./mvnw quarkus:dev -Dquarkus.profile=dev

For production (in a Docker or K8s deployment):

java -Dquarkus.profile=prod -jar target/quarkus-app/quarkus-run.jar

Or set QUARKUS_PROFILE=prod as an environment variable in your deployment YAML.

1. Add Required Dependencies in pom.xml