Basics of Kubernetes Monitoring and Logging

Every enthusiast understands the importance of Kubernetes log aggregation and log monitoring. Collecting all those diverse log files and organizing their data into an easily searchable form is where the rubber meets the road. We can do endless monitoring of our Kubernetes data but it’s all a pointless enterprise unless we’re able to aggregate those logs into a form that allows them to be of use to us. Otherwise, we’re left with a gigantic pile of unconnected puzzle pieces.

What’s the solution? In the popular open-source COS (container-orchestration system) otherwise known as Kubernetes, it’s necessary to create a logging stack. The aggregation comes in during the first phase of the operation. Specifically, when we build our Kubernetes logging stack, it will have an aggregator, a component for collecting/storing/searching, and a user-interface with alerting capabilities.

What Kubernetes Can and Can’t Do

Why all the fuss? Kubernetes is very good at doing what it does, but the core tasks of the system do not include maintaining any kind of log history. By nature, a system is extremely dynamic and only holds very recent log data. That means any prior log data is likely going to be non-existent. So, we create a logging stack to oversee all the data that flows through our Kubernetes system. It’s helpful for programmers, system administrators and even end-users to understand the core components of a Kubernetes log stack and the related stack monitoring system.

Logging Stack Basics

The entire Kubernetes logging stack is divided into three parts. This is true of most logging systems, Kubernetes-related or not. The three pieces are the log aggregator (which does the essential log aggregation tasks), the log search/storage/collector component, and the UI component (which includes an alerting function). Here’s a quick look at each of these vital pieces of the logging stack:

Log aggregator:

This part of the stack gathers up all the diverse logs from multiple pods. Many of the pods exist and run on various nodes. The aggregator simply routes all the logs to a centralized place for safe storage. Capable log aggregators share three essential qualities. They’re dynamic, efficient and extensible. In other words, you want an aggregator that is not a CPU hog, can adapt very rapidly to changes in the environment and plug into all sorts of search, storage and collection systems.

Search/storage/collector:

In addition to the obvious storage and collection functions, this part of the stack has an interface that lets us run any kind of search on the logged data we wish to perform. There are also archiving and storage management capabilities in this part of the stack.

User interface:

If you want to do any kind of log analysis, you’ll have to have a capable UI in order to search, perform queries, do debugging and, generally speaking, sift through the reams of data in front of you. Aggregating, collecting, storing and searching are immensely helpful, but without a decent UI that has an alerting function, all our work is wasted. The alerting function is typically set up alongside monitoring and logging.