Q & A Flashcards
1
Q
What is a batch?
A
A collection of messages being produced to the same topic and partition.
2
Q
What is a controller?
A
The “lead” broker within a cluster that manages the state of partitions, replicas, and partition reassignment.
3
Q
How is a controller elected?
A
When a controller goes down, the available nodes within Zookeeper will determine the controller.
4
Q
What is Zookeeper?
A
Zookeeper is a centralized service that keeps track of the brokers, configurations, topics, and partitions.
5
Q
What is a leader?
A
Partitions are owned by a single broker in the cluster and that broker is the leader. The leader is the only one responsible for produce-consumer operations.
6
Q
What are the three types of replicas?
A
Leader, follower, and preferred.
7
Q
How are new leader replicas determined?
A
When the existing leader goes down (or becomes unresponsive), if auto.leader.rebalance.enable=true is set (default) it will check if the preferred leader is in-sync and select it as the leader. Otherwise, another in-sync replica will be chosen.
8
Q
What is retention? What are the two types of retention?
A
Retention is a configurable time window that will determine how long messages are stored within a given topic. The two types of retention are delete and compact.
9
Q
What is log compaction?
A
Compaction is a type of retention in which only the latest value of a given key is retained after the retention period has elasped.
10
Q
Where is broker information stored in Zookeeper?
A
Under the /brokers/ids directory.
11
Q
What is an ephemeral node?
A
When a broker starts up, an ephemeral node is created to represent it in Zookeeper. This node will stick around to allow brokers that go offline to immediately rejoin the cluster once back online.
12
Q
What is an ensemble?
A
A cluster of Zookeeper nodes.
13
Q
What is the preferred number of Zookeeper nodes?
A
An odd number, preferably something that adheres to 2N+1 as Zookeeper requires a quorum to make elections and respond to requests.
14
Q
What is the default port(s) for Zookeeper?
A
2181 is the primary port, 2888 is used for elections, and 3888 is the leader port.
15
Q
What is the default broker port?
A
9092
16
Q
What is the default KSQL port?
A
8088
17
Q
What is the default schema registry port?
A
8081
18
Q
What is the auto.create.topics.enable setting? What actions can cause a new topic to be created?
A
When enabled, auto.create.topics.enable allows the broker to dynamically create topics if they don’t already exist. Any attempts to produce, consume, or request metadata from a topic will cause it to be created (using the default replication and partition settings from the broker).
19
Q
What is the default number of partitions?
A
One partition is the default, although it is not preferred for scaling purposes.
20
Q
How is a request handled in Kafka?
A
The process goes:
- Client Request
- Broker
- Partition Leader(s)
- Response
- Client
21
Q
Are there any guarantees within Kafka with regards to ordering of messages?
A
Messages are always guaranteed to be ordered over a single partition.
22
Q
What is a segment? What do they contain?
A
Partitions are divided into segments, which default to either 1GB of data or a week of messages. Each segment contains the messages (keys, values) over two indices (one related to offsets and another related to timestamps)
23
Q
What is the unit of storage within Kafka?
A
A partition
24
Q
What is stored within a message on disk?
A
The key, the value, a checksum for corruption, the encoding, format, timestamp.
25
When is compaction evaluated?
When a given segment of a partition is closed, any log compaction will be performed.
26
Where does Kafka store any dynamic topic configurations?
In Zookeeper
27
What producer setting determines when a given message is ready to be consumed? What are the different options for it?
The acks property determines when a message has been "received" and it can be set to the values of 0, 1, or all.
28
What do the different ack configurations mean?
If acks=0, no acknowledge is required from the broker that the message was received. If acks=1, the leader partition must confirm to have received the message. If acks=all, the topic specific min.insync.replicas property will be used to govern how many replicas have to write the message before returning a response.
29
What are the three minimally required settings for a producer to have configured?
A producer at a minimum needs a broker (e.g. bootstrap server), a key serializer, and a value serializer.
30
What is linger.ms? How can it help with batching?
linger.ms defines the time to wait before sending a batch, which can allow time for more messages to arrive and be sent, which can increase throughput.
31
What are some common mechanisms to increase producer throughput?
You want to ensure that batching is being performed effectively, so adequate min/max batch sizes, adequate linger values, and enabling compression.
32
What is the default compression used for Kafka messages? What are the other options?
By default, messages are not compressed. Supported compression types are snappy, gzip, and lz4.
33
What are the roles of the producer and consumer during compression?
The producer is responsible for compressing the messages and the consumer is responsible for decompressing them.
34
What is max.in.flight.requests.per.connection? What are the potential dangers of adjusting it?
max.in.flight.requests.per.connection is a producer setting that indicates how many messages can be sent to the server without receiving a response.
Setting this too high can result in batching becoming less efficient, increased memory usage, and error may result in loss of proper ordering.
Setting the value to 1 will ensure that only a single message is sent at a time, guaranteeing message ordering.
35
What are the two types of errors when producing a Kafka message? What are some examples of each?
Retriable and non-retriable.
Retriable messages include those were the broker or leader was not available, which will be automatically retried in hopes that a new one was elected and the message resolves.
Non-retriable messages include those that will consistently fail without some type of interaction such as a MessageSizeExceeded, which will throw an exception immediately.
36
Why are the brokers referred to as bootstrap servers?
Since each broker contains metadata about the other brokers, any of them is capable of receiving a message and sending it to the appropriate place.
37
What is unclean.leader.election.enable?
If unclean.leader.election.enable is set to true, it allows for an out-of-sync replica to be elected as leader. This is only useful if you don't necessarily care about data integrity and prefer availability over it.
38
What is a consumer group?
A group of related consumers that perform a specific task. Each consumer within a group will have mutually exclusive partitions and offsets.
39
What is a consumer?
Similar to a producer that produces messages to a broker and a specific topic, a consumer consumes messages from a given topic.
40
What is a rebalance? What can cause them?
Rebalancing is the act of changing responsibility for a given partition from one consumer to another. This can occur when the number of partitions for a topic changes or the addition/removal of a consumer from the group.
41
What are the three supported syntaxes to subscribe to a topic?
A topic can be subscribed to via a string, an array of topics, or a pattern (e.g. regular expression).
42
What is the polling loop responsible for? What happens on the initial poll?
The polling loop will handle retrieving records, metadata, partition rebalancing info, heartbeats, and more during each poll. Additionally the initial poll is responsible for registering with the group coordinator and finding the appropriate partition to use.
43
What is the session.timeout.ms property?
It’s the number of milliseconds that a consumer can be unresponsive from the broker without timing out.
44
What is auto.offset.reset? What are the supported values of it?
auto.offset.resets lets a consumer group know where to begin/continue reading offsets from for a given topic. It can be set to earliest, latest, or none (which will blow up if any existing offsets are found).
45
What is a worker?
A single Java process, generally used with regards to Kafka Connect.
46
What is the tasks.max property? What are the preferred configurations for sink connectors? What about source connectors?
This designates the number of workers that will work sending data into/out of Kafka topics.
For source connectors, it should be set to one. For sink connectors, it can be higher.
47
Where does Kafka store schemas from the schema registry?
In the __schemas topic.
48
What is a SpecificRecord? What about a GenericRecord?
A SpecificRecord is a strongly typed Java class generated via a Maven or Gradle plugin targeting existing Avro files.
A GenericRecord is a explicitly declared schema that must be accessed via index or name specifically.
49
What happens if a schema isn’t found locally for a message?
If it isn’t found in the cache, it will be requested from the Schema Registry.
50
What is a magic byte?
It’s a bytecode value that represents the schema from the schema registry that is stored in the message.
51
What are the common types of compatibility types the schema registry supports?
There are multiple: backwards (indicates a chance can be produced/consumed by the previous version of the schema), forwards (it can only be read by newer versions of the schema), full (it’s forwards and backwards compatible), and none (schema isn’t validated)
52
What is a topology?
It’s the dynamic acyclic graph that represents how a message flows through a streaming environment.
53
What’s the difference between a stream and a table?
A stream represents a full changelog whereas a table only contains the latest value of every key.
54
What language was Kafka written in? By what company?
It’s mostly written in Scala and was written by an internal team at LinkedIn.
55
What is Kafka?
It’s a distributed, resilient, real-time pub-sub system.
56
What 5 Kafka Streams operators are stateful?
The following are all stateful: aggregate, reduce, count, join, windowing.
57
What are the 11 operations in Kafka Streams are stateless?
The following operations are stateless: map, flatMap, mapValues, flatMapValues, branch, peek, groupBy, groupByKey, filter, inverseFilter, and forEach.
58
What are the four types of windows? Explain each.
Tumbling - time-based, fixed size, non-overlapping (e.g. 0-5, 5-10, 10-15)
Hopping - time-based, fixed size, overlapping (e.g. 0-5, 3-8, 6-11)
Sliding - fixed size window based on timestamp differences (only used in joins)
Session - non-time based, dynamically sized based on records (e.g. used to aggregate key based events into sessions)
59
What are the eight primitive types in Avro?
string, int, float, long, null, double, byte[], boolean
60
What are the six complex types in Avro?
record, array, fixed, enum, union, map
61
What are the optional fields for an Avro record?
doc, aliases, order, and default are all optional.
62
What are the two supported protocols for accessing the Schema Registry?
HTTP and HTTPS are the only supported protocols.
63
What occurs when you attempt to send a non-supported type to the Schema Registry?
You’ll receive a SerializationException from the registry itself.
64
What type of schema change occurs when you remove a field with a default value?
This would be a Full schema revision as it would be forwards and backwards compatible.
65
A field was added to the an schema without a default, what kind of revision is this? Why?
This is a forward revision as it’s only forwards compatible since previous versions would not have the new field.
66
What are two primarily reasons Kafka is so efficient at messaging?
It leverages Zero-Copy and thus never writes bytes directly to disk during transfer and it never transforms the data.
67
What common requirement can cause an application to lose the Zero-Copy optimization?
The use of SSL will require that messages are decrypted by consumers at each stage for consumption.
68
Can a message be modified after sending?
No. Messages in Kafka are immutable.
69
At what level is ordering guaranteed in Kafka?
At the partition level all messages are guaranteed to be in-order.
70
Why might you define a key for a message in Kafka? What change could be made that could affect that reasoning.
You can define a key in order to influence partitioning as a partitioner will always place a given key in a given partition. This doesn’t hold true if the number of partitions changes.
71
If you have multiple definitions within a configuration for retention period (e.g. hours, minutes, milliseconds), which wins?
The smallest existing unit of measure will take precedence (i.e, 120ms over 120s)
72
What is the limit for the number of replicas that can exist for a topic?
A topic cannot have more replicas than brokers exist in the cluster.
73
When might min.insync.replicas be ignored?
If acks=all is not set, this property isn’t recognized at all.
74
What producer setting can you use to remove duplicate that may have been introduced from network latency?
You can use enable.idempotence=true to help remove duplicates from being produced.
75
If a message is produced with a null key, what partition is it sent to? Conversely, what is a message is sent with a key and a null value?
If no key is provided, the message will be routed to a random partition. If a message has a key, but a null value, the message is considered a tombstone and it will signal that all messages with that key need to be removed on the next cleanup.
76
What’s the default hashing algorithm used by the Kafka partitioner?
The murmur2 algorithm.
77
Can producers be multithreaded? What about consumers?
Producers in Kafka are thread-safe, however consumers are not and should not ever be assigned multiple threads.
78
What is one of the most important consumer metrics to monitor? What can it tell you?
records-max-lag can indicate the lag over a window of time that the consumer(s) are falling behind. If this increases over time it can be a sign that the consumers are ill-equipped to keep up with producer and need to be scaled/tuned
79
Where are consumer offsets stored?
In the __consumer_offsets topic.
80
What are the three delivery semantics used by Kafka?
At-Most-Once (commits offsets after receiving but prior to processing), At-Least-Once (default, commits after processing, can allow potential dupes), Exactly-Once (can only be achieved via Kafka-Kafka)
81
What are the differences between KTables and GlobalKTables?
Both store all of the latest versions of a message (by key) however GlobalKTables load every message across all partitions, removing the need to repartition tables. They also allow you to join on non-key fields.
82
What’s a Sink Connector? Likewise, what’s a Source Connector?
A Sink connector handles importing data from Kafka into external source (e.g. a database). A Source connector doe the opposite and imports data from external sources into Kafka.
83
What are the two important timing configurations for Zookeeper? What do they do.
initLimit (how long it takes for Zookeeper to initialize with the current leader) and syncLimit (how long they can be out of sync); both of these multiples of tickTime.
84
What’s the command to find all partitions without a leader?
kafka-topics.sh --bootstrap-server localhost:9092 --describe --unavailable-partitions
85
What’s the CLI tool for consuming from Kafka topics?
kafka-console-consumer.sh
86
If you want to read from a specific partition in Kafka Streams, which function would you use?
The assign() function allows you to pass in a collection of partitions to read from. It cannot be used in conjunction with a normal subscribe call.
87
What does replication factor mean with regards to topics?
A replication factor of five would mean than each partition of the topic would live on five different brokers.
88
When running Kafka inside Kubernetes, you noticed terribly slow startup times related to restoring the state, what can you do to improve this?
Consider mounting the RocksDB as a persistent volume.
89
What’s the output of a KStream-KTable join?
A KStream.
90
In KSQL, if you want to read all of the messages for a topic, what should you do?
Set the auto.offset.reset to earliest.
91
What component in a Kafka cluster stores ACL information? Where does it store it?
It’s store in Zookeeper, under the /kafka-acl/ directory.
92
What are the three internal Kafka Connect topics?
connect-offsets (stores offsets), connect-configs (stores source connector offsets), and connect-status (elects leaders for connect)
93
A request just returned a NotLeaderForPartitionException, what should happen next?
NotLeaderForPartitionException is a retriable exception so send another request and hopefully a leader has been elected and can fulfill the request.
94
If a Zookeeper ensemble contains five nodes, what happens if two go down?
Nothing, Zookeeper will still have a quorum (2N+1) to continue to respond.
95
What two things are needed to consume from a topic?
Any of the brokers from the cluster the topic resides in and the name of the topic.
96
How are offsets committed for a topic?
A request is made to the broker responsible for the topic, sometimes calls the Global Coordinator broker, to set the offsets . They are not committed directly.
97
How is a Controller elected? What is it responsible for?
It’s elected by a Zookeeper ensemble and responsible for managing partition leader elections and all other traditional broker responsibilities.
98
What type of joins are always windowed within Kafka Streams applications?
A KStream-KStream join always requires a window.
99
When is log compaction evaluated? What determines if it’s performed?
Every time a segment (component of a partition) is closed, compaction is evaluated. If there is enough data that is considered “dirty” (via configuration), then it’s performed.
100
Where does the Schema Registry inside of a Kafka cluster?
It’s contained in a separate JVM component and interacted with via a REST interface.
101
If you have a long running process that is continually interrupted by rebalances, what can you adjust to help with this?
Adjust the max.poll.interval.ms property by a significant margin / percentage to let the broker know the process isn’t dead so it doesn’t rebalance.
102
What’s the response from a send() operation in Kafka?
A Future object that contains information related to the record, it’s offsets, partition, etc.
103
What confirmation property needs to be set to ensure all of the consumers share offsets?
You need to use the group.id to make sure that consumers are part of the same consumer group and delegate partitions accordingly.
104
What are the three supported authentication mechanisms in Kafka?
SASL/SCRAM, SSL, and SASL/GSSAPI
105
If you have a producer that can produce at 10MB/s, how would you determine your maximum throughput?
Just multiply your production rate by the number of brokers you have (or add more brokers to increase it)
106
In terms of modeling, how are streams and tables related to static and and non-static data?
Tables are generally comprised of static data, whereas streams will consist of more dynamic transactional data.
107
A customer would like all of their data sent to a specific partition, what might you do to handle this?
Consider writing a custom partitioner to route data for their key to a specific partition.
108
After failures, your consumer is taking too long to rebalance, what two settings can you adjust? How do those help the situation?
You can decrease the session.timeout.ms to help speed up the rebalancing itself and decrease the heartbeat.interval.ms to identify the inactive consumer more quickly (to trigger the rebalance)
109
Is heap size important to Kafka? How much is an adequate amount?
Not really, it uses a minimal amount and the rest is used as a page cache. 4-8GB is usually more than enough.
110
What type of joins do not require each side to be co-partitioned?
Any joins involving GlobalKTables, e.g. KStream-GlobalKTable, since they already contain all the data across all partitions.
111
What is Mirrormaker? What is it used for?
Mirrormaker can copy data across clusters and acts as a middleman for consumers/producers on each side.
112
What can calling a get() after a send() request get you? What’s the downside here?
The response of the get() call will allow you to access the metadata for the record. The downside is that this will decrease throughput since it is waiting on the response from the broker before moving on.
113
What is Active-Passive mirroring?
This is the concept that one cluster is for reading and writing (Active) while another copy is mirroring it but only used for reading purposes (Passive).
114
What do leading underscores denote with regards to topics?
This indicates they are internal topics to Kafka.
115
Define a stream?
An abstraction representing an unbounded data set.
116
What are the three types of time in Kafka?
Event Time (when the event actually occurred), Append Time (when the message arrived in Kafka), and Processing Time (when the processor received it).
117
If you have a consumer that doesn’t automatically commit offsets and you call close() what immediately happens.
Since the consumer doesn’t wait to commit offsets (they are committed manually) it will have nothing to wait for and will immediately trigger a rebalance.
118
What’s a common pattern for generated unique identifiers in Kafka?
topic + partition + offset
119
What is the broker.rack configuration? What does it do?
When setting this, assuming a proper configuration, will ensure that partitions will be spread across multiple racks.
120
What are two common exceptions that can occur on a client prior to sending a message?
A SerializationException or a BufferExhaustedException.
