Streamlining Log Analytics: Creating an Elasticsearch Cluster on a Cloud-Native Kubernetes Cluster

Managing logs is a critical component of any modern application or infrastructure stack. However, with the increasing complexity of cloud-native environments, it can be challenging to aggregate and analyze logs from multiple sources. In this article, I will explore how to streamline log analytics by creating an Elasticsearch cluster on a cloud-native Kubernetes cluster. By leveraging the scalability and resiliency of Kubernetes and the powerful search and indexing capabilities of Elasticsearch, you can create a robust log management solution that can handle large volumes of data in real-time. Let’s dive in!

Prerequisites

Kubernetes 1.19+
Helm 3.2.0+
Underlying infrastructure that supports PV provisioners

Accelerating Application Development with Azure-Ready Container Charts

This Helm chart is designed to retrieve container images from the public repository of the Azure Marketplace. To download and install all the charts from this repository, use the following command:

$ helm repo add bitnami-azure https://marketplace.azurecr.io/helm/v1/repo

Install the helm chart

Install the helm chart with name elasticsearch with Namespace elasticsearch-master

$ Kubectl create namespace elasticsearch-master
$ helm repo add elasticsearch https://charts.bitnami.com/bitnami
$ helm install elasticsearch my-repo/elasticsearch -n elasticsearch-master

After installing the helm chart run this command and verify the installed version is running perfectly

$ Kubectl get all -n elasticsearch-main

Output is as follows

If you want to setup the Kibana along with this please use below steps:

Proceed with the deployment of Kibana by installing the respective Helm chart

$ helm install kibana --version 7.17.3 elastic/kibana -n elasticsearch-master

Modifying the value of ‘healthCheckPath’ in the ‘values.yaml’ file from ‘/app/kibana’ to ‘/api/status’ can potentially resolve the error encountered during the readiness probe.

And set the elasticsearch url as follows

- env:
        - name: ELASTICSEARCH_HOSTS
          value: <url>

Setting up an Elasticsearch cluster on a cloud-native Kubernetes cluster can greatly enhance your log analytics capabilities. By following the steps outlined in this article, you can effectively streamline your log data and leverage the scalability, portability, and automation benefits of Kubernetes.

However, it’s worth noting that this is just the tip of the iceberg when it comes to Elasticsearch and Kubernetes. There are numerous other advanced features and techniques that you can explore to optimize your log analytics, such as horizontal pod autoscaling, load balancing, and security configurations.

We hope that this article has provided you with a solid foundation to get started with Elasticsearch and Kubernetes, and inspired you to further explore the powerful possibilities of this combination. With the right tools and strategies, you can gain invaluable insights into your systems and applications, and make data-driven decisions to drive your business forward.

Thank you for reading, and happy logging!

References

elasticsearch 19.5.0 · joaquinito2051/bitnami-aks