To give some background, Kafka topics are feeds where in messages are produced to and consumed from clients. It is crucial to define some naming conventions in the initial phases of the project, for proper Governance on the topics.
Topics have partitions and under the hood, there are files and folders created for every partition in the file system.
For example: if a topic name is “logstopic” and has 2 partitions (with one broker), there are folders in the file system (Kafka Broker logs directory configuration log.dirs) created for this topic like below.
If log.dirs=/usr/share/kafka, we should see 2 directories for this topic
/usr/share/kafka $ ls -ltr
- logstopic-0
- logstopic-1
Each of this directories contain files like below.
[kwuser@awsserver logstopic-0]$ ls -ltr
total 4
-rw-r — r — . 1 kwuser kwuser 3482 Oct 8 2019 00000000000000000000.log
-rw-r — r — . 1 kwuser kwuser 10485756 Jan 14 09:03 00000000000000000000.timeindex
-rw-r — r — . 1 kwuser kwuser 10485760 Jan 14 09:03 00000000000000000000.index
-rw-r — r — . 1 kwuser kwuser 8 Jan 14 09:04 leader-epoch-checkpoint
00000000000000000000.log file stores the actual events. More details on Log entries and segments : Kafka Log
Limitation on topic names
Few limitations on how a topic name can be created
- Can be Alpha numeric
- No spaces in the topic.
- minus (-) can be used in a topic name
- . (dot) and underscore(_) can be in a topic name, but better not use to avoid any metrics collision.
- Max length of 249 characters
- Bad characters : (‘/’, ‘\\’, ‘,’, ‘\u0000’, ‘:’, “\””, ‘\’’, ‘;’, ‘*’, ‘?’, ‘ ‘, ‘\t’, ‘\r’, ‘\n’, ‘=’)
Consider the below
If you think any of the below don’t change in future, consider the below in the topic names.
- Application name
- Functionality
- Feature
- Domain
- Confidentiality/Sensitivity
- Business or Logging
- Suffix
Few examples : customernotifications, systemlog, webaudit, paymentack, analytics_post, mortgage_notifications_topic
- Avoid block letters like MortgageDetailstopic
- Better all lowercase
- Avoid camel case, and follow snake case like mortgage_details_topic to make it self-explanatory
Enforcing rules :
Disable auto.create.topics.enable property on Kafka broker, so that users cannot create topics directly on Producing messages., and make sure someone or any application reviews the topic names before they getting created. Basically it has to enforce all the defined rules to follow certain guidelines. It is always better to automate this process and notify teams or Admin incase of violations.
Renaming Topic ?
It is not possible or supported by Kafka, to rename a Kafka topic.
As there are internal dependencies like consumer offset positions, offset checkpoint index etc, it is not an easy operation to rename. So as of now, only way to rename is, delete the topic and create another topic with a new name.
There is an issue raised to Kafka, but it is not going to be fixed. You may follow it. Jira Issue
However, to get back your data on the new topic, you can use Mirror Maker or Confluent replicator, and get data replicated.
Kafkawize context
Kafkawize allows users to request for topics with the above limitations mentioned, together with a Prefix configuration. In certain Kafka implementations, it is necessary to get topics created based on the environment.
Ex : devloggintopic, tstloggintopic, uatloggintopic, prdloggingtopic
If this has to be enforced, a Prefix can be configured per environment, and on requesting Kafka topics, it enforces users to apply the prefix, conforming to data integrity.
Conclusion
In this article, we learnt about how Kafka topics are created on the file system, and how certain naming conventions help in streamlining application events. As it is not possible to rename the topic, it’s better to get it the first time right.