17 Kafka interview questions

The four widely used components in Kafka are:

1. Topic - Topic refers to a stream or bundle of messages that belong to a similar topic.
2. Producer - Producers are used to communicate and publish messages on a topic.
3. Broker - Brokers are a set of servers where all the published messages are stored.

Consumer - Consumers subscribe to a particular topic and retrieve data about the messages from the Brokers.

The two commonly used methods to transfer messages are:

• Queuing - Queuing is a messaging system in which Kafka consumers can read messages from a server. However, each message goes only to a set of consumers.
• Publish-Subscribe - In this messaging system, the message is broadcasted to all the Kafka consumers.

In Kafka, every partition has a server that takes the role of a leader. Subsequently, the partition may have one or more servers that act as follower(s).

The leader performs tasks such as reading and writing to the partition. The followers passively replicate the leader. In the event that a leader fails, a follower takes up the role of the leader. This behavior helps in balancing the load on the server.

ZooKeeper is used to communicate between different nodes in a cluster. ZooKeepers are also used to commit an offset; in cases where the node fails, the data can be retrieved from the committed offset.

Apart from these roles, a ZooKeeper is also used to detect leaders, distributed synchronization, configuration management, etc.

A Kafka server can be started using the following script:

bin/Kafka-server-start.shconfig/server.properties

However, since Kafka uses ZooKeeper, we can start the ZooKeeper server using the following script.

bin/zookeeper-server-start.shconfig/zookeeper.properties

Kafka consumer groups consist of one or more consumers that are subscribed to and consume a particular topic.

This is a function that is unique to Kafka.

• Faster - Kafka uses brokers, this can be used to serve thousands of clients with high write speeds
• Scalable - Data is properly partitioned and streamlined this facilitates scalability.
• Durable - Kafka uses leaders and followers which minimizes data loss

Kafka has 4 main APIs and they are:

1. Producer API
2. Consumer API
3. Streams API
4. Connector API

The maximum message size that Kafka can receive is 1000000 bytes.

Messages in Kafka are contained in a partition and are assigned a unique ID, this ID is called an offset. This offset is used to identify messages within a partition.

ZooKeepers stores the offset of messages consumed and in case the particular message fails, it can be retrieved from the Broker.

Whenever a broker receives a message, it sends back an acknowledgement or ack to confirm that the message has been received successfully.

Ack level is used to define the number of acknowledgements that the producer requires before being considered complete.

The three essential broker configuration files are:

• Broker.id
• Log.dirs
• Zookeeper.connect

ISR stands for In-Sync Kafka; they are classified as a set of message replicates that are synced with the leaders.

The various ways that can be used to tune Apache Kafka are:

• Tuning Kafka Producers
• Tuning Kafka Brokers
• Turing Kafka Consumer