How to Make the Most Out of Kubernetes Annotations

Kubernetes (K8s) is one of the best, if not the best, container orchestration systems to date. An important part of what makes it an excellent container management technology is its powerful annotation and labeling function, which provides the ability to attach arbitrary non-identifying metadata to objects.

Metadata, consisting of labels and annotations, plays a crucial role for Kubernetes, providing its API the flexibility that developers prefer. It enables the grouping of resources, management of deployments, and the redirection of requests. It is also useful in K8s app troubleshooting. These labels and annotations are retrieved by clients such as K8s libraries and tools.

Kubernetes annotations: the basics As the two ways of attaching data to Kubernetes resources, it is essential to get acquainted with both labels and annotations. They may sound synonymous, but they are not. They have different formats and uses.

Kubernetes labels appear as a pair of key and value strings. They are used to indicate identifying attributes to objects such as pods. These attributes have some form of relevance or meaning to users, but they do not follow semantic rules in line with the core system.

Kubernetes annotations, on the other hand, also consist of a pair of key and value strings similar to labels. They are different from labels, though, because they store arbitrary non-identifying data. Also, Also, annotations are not employed in filtering, grouping, or operating on the resources. They are there to provide additional details about the K8s resources being annotated. Examples of these are the contacts of the team responsible for the software deployment, logs, and software audit information.

When to use and what to write as annotations Should annotations be used all the time? Ideally, yes. The Kubernetes website suggests that annotations are a way to provide convenient information that can help find the team or developers responsible for specific services that require troubleshooting. “Troubleshooting always begins with information gathering. While much attention has been paid to centralizing machine data (e.g., logs, metrics), much less attention has been given to the human aspect of service discovery,” the post writes.

Kubernetes annotations can be used to indicate who owns a specific service, the source for the service, the Slack channel the service owner is working on, and the issues identified and being monitored. These details are highly useful and can be annotated directly on the services with great ease.

The annotations feature of Kubernetes is often overlooked and underused. This should not be the case given how useful they are especially when it comes to troubleshooting. The information they attach to objects may be external to Kubernetes or not used to identify, group, and filter resources. However, the bits of details they bear expedite the communication between developers to quickly address problems as they are spotted.

Annotations can contain any information since it does not directly affect the structure, operation, and other core aspects of the services being annotated. However, this should not be construed to mean that anyone can just enter whatever they want as annotations. The annotations should be there to be useful, not to create information noise that confuses or inconveniences others who are viewing the data.

Best practices Here are three important reminders on using annotations:

1. Always use the correct syntax
2. Be clear with the purpose of the annotation
3. Make sure that the data and usage of the annotations and labels do not overlap.

Annotations, as mentioned, are a pair of strings: the prefix and name. However, the prefix is only optional. An annotation can be added with only the latter part. The prefix must be a DNS subdomain that is at most 253 characters with a slash in the ending part (example: k8s.kubernetesannotations.com). The name, on the other hand, should not be more than 63 characters.

If the prefix is omitted, it means that the annotations are private, which means it is only meant to be used by the user and their cluster. If both prefix and name are present, it is understood to mean that the annotations are viewable beyond the cluster. Developers should be clear if they want the annotations available to clients or if it should only be for internal consumption.

Here’s an example of a pod with two labels and two annotations.

apiVersion: v3
kind: Pod
metadata:
  name: demo
  labels:
    environment: production
    app: nginx
  annotations:
     howtousekubernetesannotations.com/owner: joe
     howtousekubernetesannotations.com/owner-phone: (555) 555-1234
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    - containerPort: 8080

Based on the details shown, the pod is a nginx app running in production created by developer Joe who can be contacted at (555) 555-1234. The annotation “howtousekubernetesannotations.com/owner: joe” or simply “owner: joe” can actually be added to the Labels section if the pods are to be grouped according to their owners. However, it is advisable not to have the “owner: joe” in both labels and annotations.

The decisive use of labels and annotations is crucial to make sure that clusters are easy to operate with automated tools. If there are conflicts with the usage and data indicated in the labels and annotations, it will be difficult to use client tools such as Helm and kubectl with the clusters.

Visualizing annotations With microservices and annotations piling up over time, it understandably becomes difficult to go over them. Using the "kubectl describe" can eventually get tedious, and it becomes impractical to provide direct access to Kubernetes cluster. One way to address this difficulty is visualization, which can be done through service catalogs.

These service catalogs are not the same as the Kubernetes Service Catalog project, though, which is intended to allow K8s operators to introduce different services to their clusters. Service catalogs here refer to the internally-facing developer portals that can be used to provide critical details about microservices. ServicesDB by Shopify is a good example of these annotation-visualizing service catalogs.

Make adding annotations a habit Annotations may not be compulsory in container orchestration, but they help add observability within microservice systems. As Kubernetes suggests, it puts in a "human service discovery" mechanism that makes it easy to contact developers when addressing problems that are better solved with the help of the owners of pods or clusters.

Annotations also provide useful details that can help in identifying and understanding issues. It would be more advantageous than bothersome to habitually add annotations with the write syntax and without usage and data overlaps.

Image: Pexels