Mark
#1
Apache Kafka is a distributed event streaming platform used to build scalable, real-time data pipelines and applications. When paired with Quarkus—a modern, Kubernetes-native Java framework—it becomes an efficient solution for high-throughput, event-driven services.
In this guide, we’ll walk you through configuring Kafka in Quarkus using application.properties. We'll use Docker for development and Kubernetes for production, keeping things clean by separating configurations with Quarkus profiles (dev and prod).

Step 1: Add Kafka and Kubernetes Dependencies

First, include the necessary dependencies in your pom.xml file: 
<dependencies>
  <!-- Kafka Messaging -->
  <dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-smallrye-reactive-messaging-kafka</artifactId>
  </dependency>

  <!-- Kubernetes Config -->
  <dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-kubernetes-config</artifactId>
  </dependency>
</dependencies>
These enable Kafka messaging support and Kubernetes ConfigMap/Secret integration for production.

Step 2: Set Up Kafka with Docker for Development

To run Kafka locally, use Docker Compose or a containerized setup like the one below: 
docker network create kafka-net
docker run -d --name zookeeper --network kafka-net \
  -e ALLOW_ANONYMOUS_LOGIN=yes \
  -p 2181:2181 bitnami/zookeeper

docker run -d --name kafka --network kafka-net \
  -e KAFKA_BROKER_ID=1 \
  -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 \
  -e ALLOW_PLAINTEXT_LISTENER=yes \
  -e KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT \
  -e KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
  -p 9092:9092 bitnami/kafka
This creates a basic Kafka setup accessible at localhost:9092.

Step 3: Configure application-dev.properties

Create a application-dev.properties file for development: 
%dev.mp.messaging.incoming.kafka-channel.connector=smallrye-kafka
%dev.mp.messaging.incoming.kafka-channel.topic=dev-topic
%dev.mp.messaging.incoming.kafka-channel.bootstrap.servers=localhost:9092
%dev.mp.messaging.incoming.kafka-channel.group.id=dev-group

%dev.mp.messaging.outgoing.kafka-out.connector=smallrye-kafka
%dev.mp.messaging.outgoing.kafka-out.topic=dev-topic
%dev.mp.messaging.outgoing.kafka-out.bootstrap.servers=localhost:9092
This config sets up an incoming Kafka consumer (kafka-channel) and an outgoing producer (kafka-out) using the local Docker Kafka broker.
To activate this profile: 
./mvnw quarkus:dev -Dquarkus.profile=dev

Step 4: Production Setup with Kubernetes

In a production environment, you’ll want to use environment variables and Kubernetes resources to avoid hardcoding connection info.

Step 4.1: Create a Kafka ConfigMap and Secret

ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-config
data:
  kafka.bootstrap.servers: kafka-service:9092
  kafka.topic: prod-topic

Secret (optional, if using auth):

apiVersion: v1
kind: Secret
metadata:
  name: kafka-secret
type: Opaque
data:
  kafka.username: cHJvZHVzZXI=       # base64 for 'produser'
  kafka.password: cHJvZHBhc3M=       # base64 for 'prodpass'

Step 4.2: Configure application-prod.properties

Now create application-prod.properties with placeholders for environment values: 
%prod.mp.messaging.incoming.kafka-channel.connector=smallrye-kafka
%prod.mp.messaging.incoming.kafka-channel.topic=${kafka.topic}
%prod.mp.messaging.incoming.kafka-channel.bootstrap.servers=${kafka.bootstrap.servers}
%prod.mp.messaging.incoming.kafka-channel.group.id=prod-group

%prod.mp.messaging.outgoing.kafka-out.connector=smallrye-kafka
%prod.mp.messaging.outgoing.kafka-out.topic=${kafka.topic}
%prod.mp.messaging.outgoing.kafka-out.bootstrap.servers=${kafka.bootstrap.servers}

%prod.quarkus.kubernetes-config.enabled=true
%prod.quarkus.kubernetes-config.config-maps=kafka-config
%prod.quarkus.kubernetes-config.secrets=kafka-secret
This setup enables Quarkus to fetch values dynamically from Kubernetes at runtime.

Step 5: Deploy to Kubernetes

In your Kubernetes deployment YAML, include the necessary environment configuration: 
containers:
  - name: quarkus-app
    image: your-app-image
    env:
      - name: QUARKUS_PROFILE
        value: prod
Quarkus will automatically activate the prod profile and pull values from the ConfigMap and Secret if mounted properly.

image quote pre code
Share: